The last year has been the Databricks year for me. New client, new stack — Spark instead of pandas, Delta Lake instead of SQL Server, notebooks instead of SSDT. And the first question I had to answer was whether the data quality discipline I'd built around Great Expectations would
The Hive Metastore is essential infrastructure in a Databricks environment. It's also a system that fails in specific, predictable ways when tables get large or partitioning is done wrong. Understanding those failure modes before you hit them is much better than diagnosing them from a production incident. The
We'd already solved the ADF side with a metadata config table and a generic pipeline. Forty-three sources, one pipeline, parameterized at runtime. The problem was what happened next. Once the raw data landed in ADLS, each source needed a Databricks notebook to clean, conform, and write it
Rubber duck debugging is one of those techniques that sounds ridiculous until you try it. The premise: explain your problem out loud to an inanimate object. A rubber duck on your desk, a stuffed animal, an empty chair. The act of articulating the problem forces you to structure it — and
The cluster is running a groupBy job. 19 tasks finish in 30 seconds. One task runs for 8 minutes. The cluster is 95% idle while one executor processes an outsized partition. This is data skew, and it's one of the most common reasons Spark jobs are slow despite
Both repartition() and coalesce() change the number of partitions in a Spark DataFrame. They are not interchangeable. Picking the wrong one doesn't cause an error — it just makes your job slower or produces unbalanced output files. Here's the distinction. The Difference repartition(n) triggers a full
The client had forty-three source systems. Forty-three databases, flat files, and APIs that all needed to land in their Azure Data Lake before anything downstream could run. Their current approach: build each ADF pipeline by hand. A senior data engineer had spent three weeks on it and was
This is the question I get most often in client engagements now. "We're starting a new data platform project. Should we use ADF or Synapse Pipelines?" I've given versions of this answer probably thirty times in the past year. Here's the complete
If you've written ETL in SQL Server, you've written MERGE. It's the standard pattern for incremental loads: insert new rows, update existing rows, optionally delete rows that disappeared from the source. Delta Lake supports the same MERGE semantics, running distributed across your cluster. Here&
A year ago I was a SQL Server and Azure SQL Data Warehouse person who had just watched Microsoft slowly back away from Azure Data Lake Analytics. Now I'm writing PySpark full-time, managing Delta Lake tables, and genuinely excited about data engineering in a way I haven&
Every year I do a rough count of how many ADF pipelines I'm actively maintaining for clients. At the end of 2020, that number is significant enough that I'd feel the loss if ADF disappeared tomorrow. That was not true at the end of 2018. Something
A notebook that you run manually is a prototype. A notebook that runs on a schedule, retries on failure, sends you an email when it breaks, and logs each run's success or failure is a pipeline. In Databricks, Jobs are the mechanism that turns the first thing into