SQL Server Agent has been the scheduling backbone of SQL-based ETL for twenty years. Multi-step jobs, success/failure routing, email alerts, retry logic, job dependencies — it's a full pipeline orchestrator built into the database engine. Moving to Databricks means finding an equivalent. Here's how the concepts
The metadata-driven pipeline generator was producing ADF pipelines and Databricks notebooks from a config table. The missing piece was infrastructure: every new client data source also needed a storage container, a Databricks cluster configuration, a linked service, and sometimes a Key Vault secret reference. Those were still being created by
Earlier versions of Great Expectations had a concept called "validation operators" — a way to bundle a validation run with actions like saving results and building Data Docs. The API worked but was awkward to configure and reason about. Checkpoints replace it with something cleaner: a named, reusable configuration
Databricks announced Delta Sharing at Data + AI Summit last month, and I've been running it in a test environment since. If you haven't seen the announcement: Delta Sharing is an open protocol for sharing live data across organizations without copying it. Recipient gets a read endpoint,
The field engineers had a problem: they were making risk decisions with data from three different ML models, each running independently, each returning a different score, and no single system aggregating those scores into a decision. One model scored credit risk. Another scored operational risk. A third — newer, still being
The standard approach to incremental file ingestion is messy: track which files you've processed in a table, query the directory listing, compare, load only the new ones, update the tracker. It works, but it's boilerplate that every pipeline ends up reimplementing. Auto Loader is Databricks'
Every pipeline that reads from Kafka eventually encounters a message it can't process. Wrong schema, missing required field, value that fails a business rule, upstream system sent malformed JSON at 2am on a Tuesday. What you do with that message determines whether your pipeline is robust or just
Seven years since ADF was in preview. Time for an honest inventory. I've been running ADF in production for the better part of that timeline. The platform has changed substantially. My view of it has changed substantially. Here's where things actually stand in 2021 -- what&
The appeal of Structured Streaming in Databricks is that it uses the same DataFrame API you already know. The query that runs as a batch job on yesterday's data can run continuously on data as it arrives, with minimal changes. This is the architectural convergence that makes it
Consumer group lag is the number of messages between where your consumer has committed its offset and where the end of the partition currently is. Zero lag means you are caught up — every message that has been published has been consumed. Rising lag means you are falling behind. Lag at
I have described the three-stage streaming architecture in a few posts now — raw ingest, deserialize, merge. This post is the implementation detail: what it actually looks like in Databricks notebooks and Workflows, with Delta Lake as the intermediate store between stages. The Target Architecture Three notebooks, three Databricks jobs, three
Databricks gives you a genuine choice between PySpark DataFrames and SQL for almost every operation. Both approaches produce identical Spark execution plans — there's no performance difference. The choice is about readability, team skill set, testability, and context. Here's how I decide. They Produce the Same Plan