All-purpose Databricks clusters are flexible — attach notebooks, run streaming jobs, train models. They're also sized and managed for those mixed workloads. If your use case is "BI analysts running SQL queries against Delta tables," there's a better-suited compute option: SQL Warehouses. What SQL Warehouses
A data catalog tells you what exists. It doesn't tell you what it means, who owns it, how sensitive it is, whether it's safe to join with another dataset, or what business rules were applied in its transformation. That context — the metadata that makes a table
Let Me Just Say It In 2022, I cannot find a compelling technical reason to choose Azure Data Factory over Synapse Pipelines for a new project, or vice versa. The feature parity is real. The Microsoft differentiation story — different tools for different use cases — is becoming marketing rather than architecture.
The logical extension of "LLM helps me write Spark code" is "LLM writes the pipeline definition from a description." If it can generate a transformation function, can it generate the Airflow DAG that orchestrates the whole thing? I've spent the last few weeks testing
Unity Catalog changed the Databricks data governance story significantly — a single metastore, fine-grained access controls, cross-workspace data sharing, and lineage tracking. It also changed the infrastructure assumptions for Great Expectations integrations, since Unity Catalog tables live in a different place than the Hive Metastore tables that earlier GE Databricks integrations
Unity Catalog's permission model is more expressive than the Hive metastore's — and more complicated. If you're coming from a Hive background where permissions were a simple GRANT SELECT ON TABLE, the UC model requires a mental shift. Here's the complete picture. The
Your Hive metastore has been accumulating tables for years. Hundreds of them, maybe more — bronze ingestion tables, silver conformance tables, gold aggregations, ad-hoc analyst tables, experiment tables that never got cleaned up. Unity Catalog is now on the horizon and you need to move them. The good news: most of
Databricks renamed Jobs to Workflows last year, and the UI redesign that came with it reflects something more fundamental: the platform has a real pipeline orchestration story now, not just a job scheduler. If you're still scheduling individual notebooks as separate jobs and managing dependencies through external orchestrators,
After creating a Unity Catalog metastore, the first practical question is: how do you give it access to your data that already lives in cloud storage? The answer involves two concepts that work together — storage credentials and external locations — and getting them right is the foundation for everything else in
Dimensional modeling gave us a set of conventions that have held up remarkably well for thirty years: fact tables at the center, dimension tables around them, foreign keys enforced by the database, constraints preventing bad data from landing. It works because the database does active work to maintain integrity. Move
The Client Question That Forces Honesty A client called me in January to talk about their new data platform. Greenfield. Azure. No legacy debt. The question: should we build on Azure Data Factory or Synapse Pipelines? Both are on the table. Give us a recommendation. I've been running
The Hive Metastore that ships with Databricks has worked fine for years as a catalog. What it has never had is governance: no column-level access control, no cross-workspace data sharing, no lineage tracking, no centralized policy management. Unity Catalog — announced in 2021 and rolling out through 2022 — is Databricks'