Testing Databricks notebooks is awkward. The testing patterns from regular Python projects (pytest, unittest, mocks, fixtures) work against local Python code, but Databricks pipelines have dependencies that don't exist outside a cluster — a running SparkSession, DBFS paths, dbutils, secrets. Most teams deal with this by not testing at
Consumer group lag is the canary in the coal mine for a Kafka pipeline under load. When lag starts rising — slowly, then faster — it usually means one of two things: the source volume increased, or the consumer is doing too much per message to keep up. The first is a
The Databricks CLI is the tool that turns a series of UI clicks into a script. If you're managing a production Databricks workspace — running jobs, deploying notebooks, rotating secrets, managing clusters — doing it through the UI is fine until you need to do it reliably, repeatedly, and without
Getting data from SQL Server into Delta Lake is one of the most common first tasks when a team starts moving to Databricks. The JDBC path works, but there are enough gotchas in the full end-to-end flow that it's worth walking through a complete, production-realistic pattern rather than
I started trying to build a metadata-driven ADF framework in 2015, with ADF v1. It was miserable. V1 didn't have ForEach. It didn't have parameters. The closest you could get was a .NET custom activity that read a config table and emitted dynamic pipeline definitions — which
I've been writing about getting tSQLt into a CI pipeline since 2013 — the principle being that tests which only run when a developer remembers to run them don't actually protect you. The same argument applies to data quality checks. An expectation suite that lives in a
Databricks just open-sourced Delta Lake last week. If you've been using it as a Databricks-internal feature (it's been available in the runtime since 2017), most of what you know still applies — the open-source release doesn't change the behavior, it changes who else can use
One of the recurring frustrations with cloud compute is startup time. You submit a job, the cluster starts provisioning, and you wait 3–8 minutes before your first line of code runs. For a 10-minute job, that's a 50% overhead. Cluster pools are the Databricks answer to this,
The native ADF Monitor view is fine for debugging a specific pipeline run. It's not fine for fleet-wide health monitoring. When I need to know "how are all my ADF pipelines doing this week?" the answer is not "open the Monitor view, click through each
DESCRIBE TABLE in SQL Server gives you columns and data types. In Delta Lake, there's a related but more detailed command called DESCRIBE DETAIL that tells you things about the physical storage of your table — information that directly affects query performance. It's one of those commands
After a few months of running Databricks in production, you develop a list of things you check when a job is slower than expected. Most of the tuning advice you'll find talks about partitioning strategy and Z-ordering, which are important — but there are a handful of configuration settings
There's a performance trap in Databricks that SQL Server DBAs fall into consistently. You know T-SQL functions — DATEPART, ISNULL, CHARINDEX, SUBSTRING. You know they're fast. So when you move to Spark SQL, you write user-defined functions for the cases where you need something the built-ins don&