Three weeks into Unity Catalog GA and I've completed two production migrations. Neither was as smooth as the quickstart documentation implies. Here's the honest breakdown of where things went cleanly and where they took longer than expected. The Easy Migrations Tables that were already external — pointing
Unity Catalog went generally available on August 18th. If you've been watching it in preview and waiting for the "should I migrate?" signal — this is it. GA means Databricks has committed to the API surface, the documentation is substantially complete, and the platform team is no
The Button That Haunted Every ADF Deployment If you've been running ADF with git integration, you know the button. The Publish button in the ADF Studio UI. The one that generates the ARM templates in the adf_publish branch. The one that every CI/CD pipeline in every
You've trained a model, logged it to MLflow, promoted it through Staging to Production in the Model Registry. Now you need something outside of Databricks to call it. A mobile app, a web service, a real-time scoring API. The path from registry entry to callable HTTP endpoint
Delta Sharing was announced at Summit last year and has been in use at various scale since. Now that Unity Catalog is generally available and Delta Sharing is integrated directly into UC, the practical setup story is cleaner than it was when I wrote about the announcement. Here's
Databricks announced Asset Bundles at Data + AI Summit last month. If you've been deploying Databricks workloads by hand — dragging notebooks through the UI, running Databricks CLI commands in CI/CD scripts, or writing custom REST API calls to deploy jobs — Asset Bundles is worth paying attention to. The
This is the follow-up to the April post about the client ingesting 1,000+ Kafka topics on a single Databricks job that could not scale. If you have not read that one, the short version: 640 vCores, 85-95% CPU, still falling behind on 200,000 messages per minute.
Three Years of Production Data Flows ADF Data Flows went GA in 2019. I've been running production Data Flows for three years. Not toy examples — production pipelines loading dimension tables with SCD logic, conformed zone aggregations, multi-source joins feeding analytical models. The kind of work where Spark
I started seriously learning Databricks in 2019 when Microsoft deprioritized Azure Data Lake Analytics. It's now mid-2022. Three years of production use, multiple clients, real production incidents, real performance tuning wins, and a few decisions I'd make differently if I were starting over. Here'
Photon launched on DBR 9.1 last November. I've been running it in production on three different workloads since then — a daily batch aggregation pipeline, a large MERGE-based SCD2 dimension load, and an ad-hoc analytical query layer that analysts run throughout the day. Six months of
A Milestone That Stopped Being a Milestone ADF's connector library crossed 100 in 2022. Microsoft announced it. There were blog posts. There was celebration. I didn't write a post about it at the time because, honestly, reaching 100 connectors is not interesting news in 2022. And
One of the friction points when adopting Great Expectations is the blank-page problem: you have a new dataset, you know you should write expectations for it, and you're not sure where to start. What ranges are reasonable for this column? What's the actual null rate?