Databricks Job Clusters vs Interactive Clusters: The Cost Math You Need to Do

SQL Server DBAs think about compute costs in one of two ways: either the server is on and you're paying for it regardless, or you're paying licensing and hardware amortized over years. Either way, you're not used to thinking about "this specific query

Delta Lake Write Modes: Append, Overwrite, Merge, and When to Use Each

When you write to a Delta table, you have four modes to choose from. Most people discover them by trial and error — they try overwrite, wonder why their data disappeared, then slowly piece together when each mode is appropriate. This post is the thing I wish I'd read

ADF Wrangling Data Flows: Power Query Inside Your Pipelines

ADF now has two types of data flows, and most people only know about one of them. Mapping Data Flows — the code-free Spark transformation engine — got all the attention in 2018. Wrangling Data Flows shipped quietly and haven't gotten nearly as much coverage. They're worth

Great Expectations SQLAlchemy Integration: Validating SQL Server Data with Python

The Great Expectations posts so far have focused on pandas DataFrames — CSV files, in-memory data, the data science use case. But most of the data I work with doesn't live in CSVs. It lives in SQL Server, or Azure SQL, or PostgreSQL. And I want to run

Installing Python Libraries in Databricks: Cluster Scope vs Notebook Scope

At some point after you've been using Databricks for a few months, the question of how to install Python libraries comes up. Not because the built-in libraries aren't comprehensive — they are — but because your work requires something specific. A data quality library, a custom connector,

The Delta Transaction Log: What Is Inside _delta_log and Why It Matters

Every Delta Lake table has a directory called _delta_log sitting alongside its data files. Most tutorials tell you it exists and leave it at that. Understanding what's actually in it changes how you reason about Delta's behavior — the ACID guarantees, the time travel, and the

Delta Lake VACUUM: The Maintenance Task Nobody Tells You About

When you run an UPDATE or DELETE on a Delta table, the old version of the data doesn't disappear. Delta writes new Parquet files with the changed rows and marks the old files as removed in the transaction log — but the old files stay on disk. That'

Delta Lake Table History: What DESCRIBE HISTORY Actually Tells You

One of the first questions I get from SQL Server DBAs when they start using Delta Lake is some version of: "How do I know what happened to this table?" In SQL Server, the answer is usually "it depends" — you might have CDC enabled, or an

Notebook Widgets: Making Databricks Notebooks Feel More Like Parameterized Queries

SQL Server DBAs know parameterized stored procedures. You pass in a date range, an environment name, a customer ID, and the procedure does the right thing based on those inputs. Logic lives in code, inputs come from outside. It's a clean pattern. Notebooks don't work that

Databricks Secrets: Stop Hardcoding Credentials in Your Notebooks

There's a pattern I see on almost every team that's new to Databricks. It looks like this: jdbc_url = "jdbc:sqlserver://prod-sql.mycompany.com:1433;databaseName=Finance" username = "svc_databricks" password = "Pr0duction_P@ssword_2018!" # please don't

ADF Mapping Data Flows GA: Six Months Into Real Production

Mapping Data Flows went generally available in late 2018. I've been running them in production for real client workloads since the preview. Six months of real data, real failures, real optimizations. Here's what I've learned. What Works Well The Transformation Set Is Comprehensive After

dbutils.fs: File System Operations When Your Storage Is Not a File System

When you're used to SQL Server, your data lives in a known location: a database, on a server, at a port. When you move to Databricks, your data lives in object storage — S3, ADLS, or GCS — which looks like a filesystem if you squint but behaves differently enough