Databricks just open-sourced Delta Lake last week. If you've been using it as a Databricks-internal feature (it's been available in the runtime since 2017), most of what you know still applies — the open-source release doesn't change the behavior, it changes who
One of the recurring frustrations with cloud compute is startup time. You submit a job, the cluster starts provisioning, and you wait 3–8 minutes before your first line of code runs. For a 10-minute job, that's a 50% overhead. Cluster pools are the Databricks answer to
The native ADF Monitor view is fine for debugging a specific pipeline run. It's not fine for fleet-wide health monitoring. When I need to know "how are all my ADF pipelines doing this week?" the answer is not "open the Monitor view, click through
DESCRIBE TABLE in SQL Server gives you columns and data types. In Delta Lake, there's a related but more detailed command called DESCRIBE DETAIL that tells you things about the physical storage of your table — information that directly affects query performance. It's one of those commands
After a few months of running Databricks in production, you develop a list of things you check when a job is slower than expected. Most of the tuning advice you'll find talks about partitioning strategy and Z-ordering, which are important — but there are a handful of configuration
There's a performance trap in Databricks that SQL Server DBAs fall into consistently. You know T-SQL functions — DATEPART, ISNULL, CHARINDEX, SUBSTRING. You know they're fast. So when you move to Spark SQL, you write user-defined functions for the cases where you need something the
The first version of almost every streaming pipeline is a single job: read from Kafka, deserialize, apply business logic, write to the serving layer. This is fine at low volume. It starts breaking down around the point where a single cluster cannot keep up with the throughput — and the breakdown
I've been building parameterized ADF v2 pipelines since the feature landed in preview. Two years of production workloads, multiple clients, dozens of factory deployments. Some patterns have survived. Some haven't. Here are the ones that hold up. Pattern 1: Generic Ingest Pipeline The foundational metadata-driven
Schema evolution in Delta Lake is one of those features that sounds like it makes your life easier — and it does — until you realize it can also silently change what your downstream queries return. Understanding the mechanics saves you from finding out the hard way. Schema Enforcement by Default By
In SQL Server, you call a stored procedure from another stored procedure with EXEC. In Databricks, you call a notebook from another notebook with dbutils.notebook.run(). The concept is the same — modular, reusable units of logic that you orchestrate into a pipeline. The implementation is different enough to matter.
Something is happening at Microsoft. I've been watching it since early 2019 and it's becoming harder to ignore. ADF and Power BI are converging. Not just in marketing language — in actual product architecture. Let me tell you what I'm seeing and what I think
If you're building data pipelines in Databricks and you're not a data scientist, MLflow might look like it's not for you. It's in the name: machine learning flow. But the core capability — tracking what ran, with what parameters, and what it produced