Three Years Into Databricks: What T-SQL Habits I Kept and What I Had to Replace

I started seriously learning Databricks in 2019 when Microsoft deprioritized Azure Data Lake Analytics. It's now mid-2022. Three years of production use, multiple clients, real production incidents, real performance tuning wins, and a few decisions I'd make differently if I were starting over. Here's

Photon in Production: Six Months of Real Query Performance Data

Photon launched on DBR 9.1 last November. I've been running it in production on three different workloads since then — a daily batch aggregation pipeline, a large MERGE-based SCD2 dimension load, and an ad-hoc analytical query layer that analysts run throughout the day. Six months of production data

ADF Connector Count Crosses 100: The Library Is No Longer the Story

A Milestone That Stopped Being a Milestone ADF's connector library crossed 100 in 2022. Microsoft announced it. There were blog posts. There was celebration. I didn't write a post about it at the time because, honestly, reaching 100 connectors is not interesting news in 2022. And

Great Expectations Profiler: Auto-Generating Expectations from Your Data

One of the friction points when adopting Great Expectations is the blank-page problem: you have a new dataset, you know you should write expectations for it, and you're not sure where to start. What ranges are reasonable for this column? What's the actual null rate? What

Data Modeling Best Practices for Lakehouse Architectures: What Dimensional Modeling Gets Right

Ten years of building data models across different platforms has given me some strong opinions about what works in a lakehouse environment versus what you'll fight against. The lakehouse relaxes a lot of the old constraints — no referential integrity, no NOT NULL enforcement at the engine level, schema-on-read

Cross-Cloud Data Engineering: Running the Same Metadata Config on Azure, AWS, and GCP

The metadata-driven pipeline design I've been building toward solves a specific problem: how do you build a pipeline that runs everywhere without rewriting it for each environment? The answer is that the pipeline doesn't know where it's running. The config table knows, and the

Unity Catalog Lineage: What It Tracks and What You Still Need to Instrument Yourself

Unity Catalog tracks lineage automatically for SQL queries and DataFrames run in notebooks and jobs attached to a UC metastore. You run a query, UC records which tables were read and which were written, and that information shows up in the Catalog Explorer lineage graph. For straightforward table-to-table transformations, this

Unity Catalog Column-Level Security: Row Filters and Column Masks in Practice

Unity Catalog's permission model handles table-level and schema-level access cleanly. What it also gives you — that Hive metastore doesn't — is column-level security: the ability to restrict specific columns within a table, and row-level security through dynamic row filters. If you're dealing with PII, financial

Delta Live Tables in Production: What Changes When You Let Databricks Own the Graph

Delta Live Tables went GA in February. I've been running it in production for a couple of months now, and the experience is different from the preview. The framework has stabilized, the documentation has caught up to reality, and I have enough production hours on it to tell

The Metadata-Driven Data Management Pattern: One Config Table, Every Client Source

The pattern I keep coming back to on client engagements is one that looks the same no matter which cloud, which tooling stack, or which domain. A config table drives everything: what sources exist, how to extract from them, where to land the data, and what transformations to apply. The

1,000 Kafka Topics, One Databricks Job, Zero Success: A Case Study in Streaming Architecture

A client came to me with a straightforward-sounding problem. They had a Kafka publisher producing over 1,000 topics — one per data source in their manufacturing network — and they needed to ingest all of them into their Databricks environment and make the data available in gold layer Delta tables for

ADF Managed Airflow: Reading Between the Lines of This Announcement

The Announcement Microsoft announced in late 2022 that ADF Studio would support authoring and running Apache Airflow DAGs on managed Airflow infrastructure. You write DAGs in the ADF Studio interface. Microsoft runs the Airflow scheduler and workers. No Airflow cluster to provision or operate. On the surface: a convenient managed