Pipeline Architecture for High-Volume Kafka: When One Job Is Not Enough

The first version of almost every streaming pipeline is a single job: read from Kafka, deserialize, apply business logic, write to the serving layer. This is fine at low volume. It starts breaking down around the point where a single cluster cannot keep up with the throughput — and the breakdown

Parameterized ADF Pipelines in Production: The Patterns That Actually Hold Up

I've been building parameterized ADF v2 pipelines since the feature landed in preview. Two years of production workloads, multiple clients, dozens of factory deployments. Some patterns have survived. Some haven't. Here are the ones that hold up. Pattern 1: Generic Ingest Pipeline The foundational metadata-driven pattern:

Delta Lake Schema Evolution: What Actually Happens to Your Existing Table When You Add a Column

Schema evolution in Delta Lake is one of those features that sounds like it makes your life easier — and it does — until you realize it can also silently change what your downstream queries return. Understanding the mechanics saves you from finding out the hard way. Schema Enforcement by Default By

dbutils.notebook: Orchestrating Databricks From Databricks

In SQL Server, you call a stored procedure from another stored procedure with EXEC. In Databricks, you call a notebook from another notebook with dbutils.notebook.run(). The concept is the same — modular, reusable units of logic that you orchestrate into a pipeline. The implementation is different enough to matter.

ADF and Power BI: Reading the Product Roadmap Tea Leaves

Something is happening at Microsoft. I've been watching it since early 2019 and it's becoming harder to ignore. ADF and Power BI are converging. Not just in marketing language — in actual product architecture. Let me tell you what I'm seeing and what I think

MLflow for Pipeline Engineers: Tracking Experiments Without Becoming a Data Scientist

If you're building data pipelines in Databricks and you're not a data scientist, MLflow might look like it's not for you. It's in the name: machine learning flow. But the core capability — tracking what ran, with what parameters, and what it produced

Data Docs: Auto-Generated Data Quality Reports from Your Expectation Suite

One of the quieter but more useful features in Great Expectations is Data Docs — automatically generated HTML documentation that shows you, in a human-readable format, what your expectations are and whether your data passes them. It's not something you build; it's something GE generates from the

From Kafka to Delta Lake: The Streaming Pattern That Replaced My SQL Pipeline

For most of my career, a "real-time pipeline" meant reading from a SQL Server table on a short schedule — every 5 minutes, every minute, as fast as you could run the query. It worked, but it had edge cases: what if the job ran slower than the interval?

TabNine and the First AI Completion That Actually Helped

I've been burned by AI code completion enough times to approach every new entrant in this space with genuine skepticism. IntelliSense completes your method names. Kite autocompletes Python idioms based on what's in scope. These are useful but they're not surprising — they're

Delta Lake Concurrent Writes: What the Transaction Log Does When Two Jobs Collide

Delta Lake's transaction log is designed to handle concurrent operations safely — multiple readers can run alongside a writer, and the writer's changes become visible atomically. But what happens when two jobs try to write to the same Delta table at the same time? The answer is

Structured Streaming Watermarks: Handling Late-Arriving Data in Your Pipeline

Structured Streaming is how you build real-time pipelines in Databricks. You've probably already seen basic examples — read from Kafka, write to Delta. What those examples almost always skip is watermarking, and that omission tends to cause subtle problems with aggregations over event time. Why Watermarks Exist When you&

Databricks Connect: Testing Spark Code Without Spinning Up a Cluster Session

Spark jobs have a startup cost. The cluster needs to provision, the JVMs need to start, the session needs to initialize. For a two-hour batch job, that's noise. For a 45-second unit test, that's the entire runtime budget. Databricks Connect lets you run Spark code from