Last time, I compared SQL Server storage to Databricks storage. This time, let’s compare SQL Server compute to Databricks compute. How SQL Server Processes a Query When you submit your workload to SQL Server, the engine will first parse it to ensure it’s syntactically correct. If it’s not, then it..
Understanding how Databricks works with storage can go a long way toward improving your use of Databricks. Storage In SQL Server, we store our data in two highly structured file types. The first type of file is a data file; these have an extension of MDF or NDF. You get...
I’ve been a Microsoft Data professional for over 20 years. Most of that time I’ve spent in the SQL Server stack, the core query engine, SSIS, SSRS, and a little SSAS. But times changed, and the business problems grew more complex. As they did, I looked at other technologies to...
I have been writing Kafka-based pipelines since 2012. The tooling in 2022 is dramatically better — Delta Lake, Databricks Auto Loader, Unity Catalog, Schema Registry, the Event Hubs Kafka protocol. The architecture principles, on the other hand, have changed remarkably little. The things that made pipelines fail in 2013 still
Every time I connect to a new data source, the first hour is the same: exploratory queries to understand what's actually in there. How many rows, what columns, what are the null rates, does that "primary key" actually have duplicates, what does a typical row look
On a recent engagement, I was asked to provide best practices. I realized that many of the best practices hadn’t been collected here, so it’s time I fix that. The client was early in their journey of adopting Databricks as their data engine, and a lot of the development they...
Reading the Signals I don't usually make predictions in public. But the signals I'm reading in 2022 are loud enough that staying quiet feels like an abdication. Microsoft is building a unified analytics platform. They're going to announce it with a new name, probably
Every GPT-3 Playground session starts cold. The model has no memory of the schema you described last Tuesday, no recollection of the partition strategy decision you worked through together last month, no awareness that you've already ruled out three approaches and the fourth is what you settled
I wrote about the raw zone back in 2013, when the pattern was just becoming standard practice in the teams I worked with. The technology has changed significantly — Delta Lake, Auto Loader, Schema Registry, Unity Catalog — but every time I revisit the argument, I find the core principle holds. Land
Databricks Repos has been available since 2021 and is fully integrated with Git. If you're still managing notebooks by exporting them as .dbc files, checking them into Git manually, or using a sync script — stop. Repos is the right answer for notebook version control, and it's
I've been building config-driven pipelines since 2015. The pattern: a structured JSON or YAML file describes the data — source paths, target paths, column mappings, business rules — and a generic Spark job reads the config at runtime and executes it. The logic lives in the job; the knowledge
Five years of using Great Expectations across different projects — pandas-based data science, SQL Server pipelines, Databricks at scale — has produced a set of architectural opinions I keep coming back to. This post is the patterns that have held up, and the mistakes that taught me to adopt them. Maintain