Schema evolution in Delta Lake is one of those features that sounds like it makes your life easier — and it does — until you realize it can also silently change what your downstream queries return. Understanding the mechanics saves you from finding out the hard way. Schema Enforcement by Default By
In SQL Server, you call a stored procedure from another stored procedure with EXEC. In Databricks, you call a notebook from another notebook with dbutils.notebook.run(). The concept is the same — modular, reusable units of logic that you orchestrate into a pipeline. The implementation is different enough to matter.
Something is happening at Microsoft. I've been watching it since early 2019 and it's becoming harder to ignore. ADF and Power BI are converging. Not just in marketing language — in actual product architecture. Let me tell you what I'm seeing and what I think
If you're building data pipelines in Databricks and you're not a data scientist, MLflow might look like it's not for you. It's in the name: machine learning flow. But the core capability — tracking what ran, with what parameters, and what it produced
One of the quieter but more useful features in Great Expectations is Data Docs — automatically generated HTML documentation that shows you, in a human-readable format, what your expectations are and whether your data passes them. It's not something you build; it's something GE generates from the
For most of my career, a "real-time pipeline" meant reading from a SQL Server table on a short schedule — every 5 minutes, every minute, as fast as you could run the query. It worked, but it had edge cases: what if the job ran slower than the interval?
I've been burned by AI code completion enough times to approach every new entrant in this space with genuine skepticism. IntelliSense completes your method names. Kite autocompletes Python idioms based on what's in scope. These are useful but they're not surprising — they're
Delta Lake's transaction log is designed to handle concurrent operations safely — multiple readers can run alongside a writer, and the writer's changes become visible atomically. But what happens when two jobs try to write to the same Delta table at the same time? The answer is
Structured Streaming is how you build real-time pipelines in Databricks. You've probably already seen basic examples — read from Kafka, write to Delta. What those examples almost always skip is watermarking, and that omission tends to cause subtle problems with aggregations over event time. Why Watermarks Exist When you&
Spark jobs have a startup cost. The cluster needs to provision, the JVMs need to start, the session needs to initialize. For a two-hour batch job, that's noise. For a 45-second unit test, that's the entire runtime budget. Databricks Connect lets you run Spark code from
Writing a Spark Structured Streaming job that reads from Kafka and writes to Parquet files sounds straightforward until you watch it crash mid-write and leave your output directory in an ambiguous state. Did it write all the records for that micro-batch? Half of them? None? The checkpoint committed, but the
It's been a year since ADF git integration shipped. My team has been living with it — daily authoring, PRs, deployments — long enough that the novelty has worn off and the real picture is clear. Here's the honest 12-month assessment. What's Genuinely Good No More