Your SQL Skills Are More Transferable to Databricks Than You Think

Great Expectations and pandas: Validating DataFrames Before They Hit the Database

My data science project this year has me working with NOAA weather data — storm event records going back decades, delivered as CSVs, with all the consistency problems that implies. Field names that changed between years. Magnitude values stored as strings in some vintages and floats in others. Missing state codes.

Data Types in Spark SQL: The Translation Guide SQL Server DBAs Actually Need

On a recent project, we had a pipeline that worked perfectly in SQL Server, imported cleanly into Databricks, and produced subtly wrong results for six weeks before anyone noticed. The culprit: a DATETIME column that became a TIMESTAMP in Spark SQL, and a timezone assumption that was implicit in our

ADF CI/CD with ARM Templates: Shipping Pipeline Changes Like Real Code

Git integration landed. Now let's talk about what you do with it. Because "pipelines in git" is table stakes — what you actually want is a deployment pipeline that takes your ADF changes from development to production the same way you ship application code. Here's

The Spark Execution Model for SQL Pros: What Really Happens When You Run a Query

You've spent years reading execution plans. You know what a nested loop join costs. You can spot a missing index from the fat red arrow pointing the wrong direction. When something is slow in SQL Server, you click "Include Actual Execution Plan" and the answer is

Streaming Data Quality: Catching Schema Changes Before They Corrupt the Gold Layer

In batch ETL, schema changes are annoying but manageable. You run a load, the job fails, you investigate, you fix the mapping, you re-run. The damage is limited: at worst, one load cycle's worth of data is missing or wrong, and you fix it before it reaches the

Dead Letter Patterns for Streaming: Where Bad Messages Go

The scenario: your Kafka consumer is humming along, processing thousands of messages per hour, when it hits one that does not match the expected schema. The JSON has a field with the wrong type. The Avro deserialization throws an exception. Maybe the message is simply empty bytes. What happens next

Event Hubs with the Kafka Protocol: Managed Infrastructure, Zero Code Changes

Azure Event Hubs Kafka protocol support went generally available this year, and it is a bigger deal than most of the announcement posts made it sound. The practical implication: any Kafka producer or consumer can connect to Event Hubs by changing a connection string. No SDK swap. No code changes.

ADF Git Integration Has Arrived: Let's Talk About What It Does and Doesn't Do

Four years. That's how long I've been asking for git integration in Azure Data Factory. Since 2014, every ADF project I've touched has had the same operational hazard: someone makes an emergency fix in the portal, doesn't document it, and now your