I wrote about the raw zone back in 2013, when the pattern was just becoming standard practice in the teams I worked with. The technology has changed significantly — Delta Lake, Auto Loader, Schema Registry, Unity Catalog — but every time I revisit the argument, I find the core principle holds. Land
Reading the Signals I don't usually make predictions in public. But the signals I'm reading in 2022 are loud enough that staying quiet feels like an abdication. Microsoft is building a unified analytics platform. They're going to announce it with a new name, probably
Every GPT-3 Playground session starts cold. The model has no memory of the schema you described last Tuesday, no recollection of the partition strategy decision you worked through together last month, no awareness that you've already ruled out three approaches and the fourth is what you settled on.
Databricks Repos has been available since 2021 and is fully integrated with Git. If you're still managing notebooks by exporting them as .dbc files, checking them into Git manually, or using a sync script — stop. Repos is the right answer for notebook version control, and it's
I've been building config-driven pipelines since 2015. The pattern: a structured JSON or YAML file describes the data — source paths, target paths, column mappings, business rules — and a generic Spark job reads the config at runtime and executes it. The logic lives in the job; the knowledge about
Five years of using Great Expectations across different projects — pandas-based data science, SQL Server pipelines, Databricks at scale — has produced a set of architectural opinions I keep coming back to. This post is the patterns that have held up, and the mistakes that taught me to adopt them. Maintain a
Three weeks into Unity Catalog GA and I've completed two production migrations. Neither was as smooth as the quickstart documentation implies. Here's the honest breakdown of where things went cleanly and where they took longer than expected. The Easy Migrations Tables that were already external — pointing
Unity Catalog went generally available on August 18th. If you've been watching it in preview and waiting for the "should I migrate?" signal — this is it. GA means Databricks has committed to the API surface, the documentation is substantially complete, and the platform team is no
The Button That Haunted Every ADF Deployment If you've been running ADF with git integration, you know the button. The Publish button in the ADF Studio UI. The one that generates the ARM templates in the adf_publish branch. The one that every CI/CD pipeline in every
You've trained a model, logged it to MLflow, promoted it through Staging to Production in the Model Registry. Now you need something outside of Databricks to call it. A mobile app, a web service, a real-time scoring API. The path from registry entry to callable HTTP endpoint is
Delta Sharing was announced at Summit last year and has been in use at various scale since. Now that Unity Catalog is generally available and Delta Sharing is integrated directly into UC, the practical setup story is cleaner than it was when I wrote about the announcement. Here's
This is the follow-up to the April post about the client ingesting 1,000+ Kafka topics on a single Databricks job that could not scale. If you have not read that one, the short version: 640 vCores, 85-95% CPU, still falling behind on 200,000 messages per minute. The fix