I want to give you a specific benchmark. Not vague "significantly faster" marketing language — an actual result from a real workload that I measured, on identical hardware, and can explain mechanically. Here it is: geocoding 23 years of NOAA storm event radar data, C# with Parallel.ForEach beat
Here is the most common anti-pattern I see in data pipeline code: the pipeline knows the names of its tables. It knows the column names it's going to transform. It knows the business rules. It knows the output schema. All of this — structural knowledge and business logic — lives
I wrote my first tSQLt test in late 2012. Three years later, I've used it across multiple projects, introduced it to teams that had never tested SQL before, and hit most of the sharp edges the framework has. It's time for an honest assessment. What tSQLt
By mid-2015, the Azure analytical data platform had two serious options: Azure Data Lake Analytics with U-SQL, and Azure SQL Data Warehouse (Azure DW). I was using both in different client contexts and kept getting asked which one to use for a given problem. The answer wasn't "
If you've done SSIS development at scale, you know BIML. You define a config table — source tables, target tables, column mappings, load patterns — and BIML generates N perfectly isolated SSIS packages at compile time. Each package is its own executable, its own set of connection managers, its own
Airflow open-sourced out of Airbnb in June 2015. I've been running it internally for a few weeks now and I want to explain the core idea, because it's easy to look at the setup overhead and dismiss it as "Luigi with a web UI."
The three-stage streaming pipeline is not a new idea — it is the streaming equivalent of a well-understood batch ETL pattern. In batch, you stage raw files before loading them. In streaming, you land raw messages before deserializing them. The principle is the same; the implementation details differ because the data
One of the recurring costs in a Kafka-based pipeline is writing the glue code. You have a source — a database, an API, a file drop — and you need to get its data into a topic. Writing a producer from scratch means handling connection management, offset tracking, error recovery, and schema
If you are doing data engineering in an Azure shop and you have been eyeing Kafka, Microsoft has an answer for you: Azure Event Hubs. It shipped into general availability earlier this year and it is worth understanding both what it is and where it differs from the open-source alternative.
Running Spark on EMR yourself sounds like freedom. You pick the instance types, you tune the Spark config, you control the versions, you pay only for what you use. What the pitch leaves out: all of that is also your problem at 2 a.m. when a job fails due
A pipeline failed at 3 AM. The client calls at 9 AM asking why their morning report is empty. I open the ADF monitor. I see a red slice. The error message says "SqlException: Timeout expired." That's it. No row counts, no which activity failed first,
The previous post was about the geocoding job that produced zero rows. This post is about the broader problem it revealed: ADLA's monitoring story in 2015 was inadequate for detecting jobs that ran correctly but did the wrong thing. Here's what ADLA showed you, what it