If you've written ETL in SQL Server, you've written MERGE. It's the standard pattern for incremental loads: insert new rows, update existing rows, optionally delete rows that disappeared from the source. Delta Lake supports the same MERGE semantics, running distributed across your cluster. Here&
A year ago I was a SQL Server and Azure SQL Data Warehouse person who had just watched Microsoft slowly back away from Azure Data Lake Analytics. Now I'm writing PySpark full-time, managing Delta Lake tables, and genuinely excited about data engineering in a way I haven'
Every year I do a rough count of how many ADF pipelines I'm actively maintaining for clients. At the end of 2020, that number is significant enough that I'd feel the loss if ADF disappeared tomorrow. That was not true at the end of 2018. Something
A notebook that you run manually is a prototype. A notebook that runs on a schedule, retries on failure, sends you an email when it breaks, and logs each run's success or failure is a pipeline. In Databricks, Jobs are the mechanism that turns the first thing into
Delta Lake tables accumulate small files fast. Every streaming write that commits a micro-batch produces files. Every append job that runs hourly produces files. Every MERGE that touches 1,000 rows produces files. After a few weeks of steady writes, you have a table with thousands of tiny Parquet files,
SQL Server DBAs learn early to read execution plans. That skill transfers to Spark — the concepts are the same even if the output looks different. This post is the EXPLAIN decoder ring for T-SQL practitioners. Getting the Plan df = spark.read.parquet("abfss://container@storage.dfs.core.windows.net/
For two years I've been running Databricks pipelines as separate notebooks orchestrated by dbutils.notebook.run() or scheduled individually via the Jobs UI. It works, but it shows in the monitoring — five separate jobs in the UI, no unified view of the pipeline, no easy way to see
PySpark is the right tool for complex transformations, multi-step pipelines, and anything that benefits from the full DataFrame API. But if your goal is "run a SQL query against a Delta table and see the results," you don't need Python at all. Databricks SQL gives you
Delta Lake's ACID guarantees work well for standard write patterns — appends, overwrites, MERGEs. The concurrency model gets more interesting when you have multiple writers, or when you're running operations like DELETE and UPDATE simultaneously with ongoing streaming appends. Understanding the isolation model tells you what'
Most joins in Spark require a shuffle: both sides get redistributed so that matching rows from each side end up on the same executor. For two large tables, that's unavoidable. But if one side of the join is small enough to fit in memory, there's a
I pay attention to how Microsoft talks about ADF in their marketing materials. Not because I trust marketing, but because product positioning is a leading indicator of roadmap decisions, and roadmap decisions affect the architectures I'm building for clients today. The signal I've been tracking in
I've been writing about tSQLt since 2012 and Great Expectations since 2017. I've used both across different clients and projects, and I get asked regularly: which one should I use? The answer is almost always: both, depending on where in your stack you are. This post