Ten years of building data models across different platforms has given me some strong opinions about what works in a lakehouse environment versus what you'll fight against. The lakehouse relaxes a lot of the old constraints — no referential integrity, no NOT NULL enforcement at the engine level, schema-
The metadata-driven pipeline design I've been building toward solves a specific problem: how do you build a pipeline that runs everywhere without rewriting it for each environment? The answer is that the pipeline doesn't know where it's running. The config table knows, and
Unity Catalog tracks lineage automatically for SQL queries and DataFrames run in notebooks and jobs attached to a UC metastore. You run a query, UC records which tables were read and which were written, and that information shows up in the Catalog Explorer lineage graph. For straightforward table-to-table
Unity Catalog's permission model handles table-level and schema-level access cleanly. What it also gives you — that Hive metastore doesn't — is column-level security: the ability to restrict specific columns within a table, and row-level security through dynamic row filters. If you're
Delta Live Tables went GA in February. I've been running it in production for a couple of months now, and the experience is different from the preview. The framework has stabilized, the documentation has caught up to reality, and I have enough production hours on it to tell
The pattern I keep coming back to on client engagements is one that looks the same no matter which cloud, which tooling stack, or which domain. A config table drives everything: what sources exist, how to extract from them, where to land the data, and what transformations to apply. The
The Announcement Microsoft announced in late 2022 that ADF Studio would support authoring and running Apache Airflow DAGs on managed Airflow infrastructure. You write DAGs in the ADF Studio interface. Microsoft runs the Airflow scheduler and workers. No Airflow cluster to provision or operate. On the surface: a convenient managed
A client came to me with a straightforward-sounding problem. They had a Kafka publisher producing over 1,000 topics — one per data source in their manufacturing network — and they needed to ingest all of them into their Databricks environment and make the data available in gold layer Delta tables
All-purpose Databricks clusters are flexible — attach notebooks, run streaming jobs, train models. They're also sized and managed for those mixed workloads. If your use case is "BI analysts running SQL queries against Delta tables," there's a better-suited compute option: SQL Warehouses. What
A data catalog tells you what exists. It doesn't tell you what it means, who owns it, how sensitive it is, whether it's safe to join with another dataset, or what business rules were applied in its transformation. That context — the metadata that makes a table
The logical extension of "LLM helps me write Spark code" is "LLM writes the pipeline definition from a description." If it can generate a transformation function, can it generate the Airflow DAG that orchestrates the whole thing? I've spent the last few weeks testing
Let Me Just Say It In 2022, I cannot find a compelling technical reason to choose Azure Data Factory over Synapse Pipelines for a new project, or vice versa. The feature parity is real. The Microsoft differentiation story — different tools for different use cases — is becoming marketing rather than architecture.