Delta Lake OPTIMIZE: Compacting Small Files and Why It Matters

Delta Lake tables accumulate small files fast. Every streaming write that commits a micro-batch produces files. Every append job that runs hourly produces files. Every MERGE that touches 1,000 rows produces files. After a few weeks of steady writes, you have a table with thousands of tiny Parquet

Spark Execution Plans: Reading EXPLAIN Like a T-SQL DBA

SQL Server DBAs learn early to read execution plans. That skill transfers to Spark — the concepts are the same even if the output looks different. This post is the EXPLAIN decoder ring for T-SQL practitioners. Getting the Plan df = spark.read.parquet("abfss://container@storage.dfs.core.windows.

Databricks Multi-Task Jobs: Orchestrating a Pipeline Without Leaving the Platform

For two years I've been running Databricks pipelines as separate notebooks orchestrated by dbutils.notebook.run() or scheduled individually via the Jobs UI. It works, but it shows in the monitoring — five separate jobs in the UI, no unified view of the pipeline, no easy way to see

Databricks SQL: Querying Delta Tables Without Writing Python

PySpark is the right tool for complex transformations, multi-step pipelines, and anything that benefits from the full DataFrame API. But if your goal is "run a SQL query against a Delta table and see the results," you don't need Python at all. Databricks SQL gives

Delta Lake Concurrency Deep Dive: Isolation Levels, Writer Conflicts, and What Actually Blocks

Delta Lake's ACID guarantees work well for standard write patterns — appends, overwrites, MERGEs. The concurrency model gets more interesting when you have multiple writers, or when you're running operations like DELETE and UPDATE simultaneously with ongoing streaming appends. Understanding the isolation model tells you what'

Broadcast Joins in Spark: When Copying a Small Table to Every Node Is the Fastest Option

Most joins in Spark require a shuffle: both sides get redistributed so that matching rows from each side end up on the same executor. For two large tables, that's unavoidable. But if one side of the join is small enough to fit in memory, there's a

ADF Under the Analytics Umbrella: Where Microsoft Is Going With This

I pay attention to how Microsoft talks about ADF in their marketing materials. Not because I trust marketing, but because product positioning is a leading indicator of roadmap decisions, and roadmap decisions affect the architectures I'm building for clients today. The signal I've been tracking in

Great Expectations vs. tSQLt: Two Frameworks, One Principle

I've been writing about tSQLt since 2012 and Great Expectations since 2017. I've used both across different clients and projects, and I get asked regularly: which one should I use? The answer is almost always: both, depending on where in your stack you are. This post

PySpark GroupBy Aggregations: Every Aggregate You Know From T-SQL, Translated

You've been writing GROUP BY aggregations in T-SQL for years. PySpark has all the same aggregate functions — and a few you probably wished SQL Server had. Here's the translation, plus a few patterns that are unique to the distributed world. The Basic Pattern from pyspark.

Adaptive Query Execution in Spark 3.0: What the Optimizer Finally Does Automatically

Spark 3.0 shipped in June, and Databricks Runtime 7.x brought it to production workspaces. The headlining feature — Adaptive Query Execution — has been running in my pipelines for a few months now. Here's what it actually changed versus what I expected. What AQE Changed in Practice The

Checkpoint Files and Idempotency: Getting Streaming Restarts Right

Checkpoint files are the most misunderstood part of a Structured Streaming pipeline. They are not a cache. They are not a nice-to-have. They are the mechanism that makes exactly-once delivery possible. Delete them and you will either duplicate your data, gap your data, or both, depending on

The Shuffle: When Spark Redistributes Data Between Executors and Why It Costs You

If you want to understand why a Spark job is slow, you need to understand the shuffle. The shuffle is the most expensive operation in Spark, and most of the performance tuning work in any non-trivial Spark application comes back to minimizing, eliminating, or managing shuffles. What a Shuffle