Unity Catalog changed the Databricks data governance story significantly — a single metastore, fine-grained access controls, cross-workspace data sharing, and lineage tracking. It also changed the infrastructure assumptions for Great Expectations integrations, since Unity Catalog tables live in a different place than the Hive Metastore tables that earlier GE
Unity Catalog's permission model is more expressive than the Hive metastore's — and more complicated. If you're coming from a Hive background where permissions were a simple GRANT SELECT ON TABLE, the UC model requires a mental shift. Here's the complete picture. The
Your Hive metastore has been accumulating tables for years. Hundreds of them, maybe more — bronze ingestion tables, silver conformance tables, gold aggregations, ad-hoc analyst tables, experiment tables that never got cleaned up. Unity Catalog is now on the horizon and you need to move them. The good news: most
Databricks renamed Jobs to Workflows last year, and the UI redesign that came with it reflects something more fundamental: the platform has a real pipeline orchestration story now, not just a job scheduler. If you're still scheduling individual notebooks as separate jobs and managing dependencies through external orchestrators,
After creating a Unity Catalog metastore, the first practical question is: how do you give it access to your data that already lives in cloud storage? The answer involves two concepts that work together — storage credentials and external locations — and getting them right is the foundation for everything else in
Dimensional modeling gave us a set of conventions that have held up remarkably well for thirty years: fact tables at the center, dimension tables around them, foreign keys enforced by the database, constraints preventing bad data from landing. It works because the database does active work to maintain integrity. Move
The Client Question That Forces Honesty A client called me in January to talk about their new data platform. Greenfield. Azure. No legacy debt. The question: should we build on Azure Data Factory or Synapse Pipelines? Both are on the table. Give us a recommendation. I've been running
The Hive Metastore that ships with Databricks has worked fine for years as a catalog. What it has never had is governance: no column-level access control, no cross-workspace data sharing, no lineage tracking, no centralized policy management. Unity Catalog — announced in 2021 and rolling out through 2022 — is
Unity Catalog has been in preview since mid-2021. If you've been watching it and wondering when to actually engage with it, the answer is: now is a reasonable time to start experimenting, especially if you have a new workspace or a non-critical environment where you can
Delta Live Tables has been in preview for a few months and I've been watching it closely. The concept is appealing: instead of writing notebook code that explicitly manages the read-transform-write lifecycle, you declare what the data should look like at each stage and DLT figures
After seven years of running ADF in production, my mental model of what ADF is has settled. This clarity resolves most of the architectural debates I see teams have about ADF versus other tools. Let me put it plainly. ADF is an orchestrator. It coordinates the execution of tasks across
Dimensional modeling works well when your business questions are known in advance and your schema can be designed around them. It falls apart when the questions change, when you need full historical auditability of every change to every record, or when you're building a centralized enterprise data warehouse