If you've done SSIS development at scale, you know BIML. You define a config table — source tables, target tables, column mappings, load patterns — and BIML generates N perfectly isolated SSIS packages at compile time. Each package is its own executable, its own set of connection managers, its own
Airflow open-sourced out of Airbnb in June 2015. I've been running it internally for a few weeks now and I want to explain the core idea, because it's easy to look at the setup overhead and dismiss it as "Luigi with a web UI.
The three-stage streaming pipeline is not a new idea — it is the streaming equivalent of a well-understood batch ETL pattern. In batch, you stage raw files before loading them. In streaming, you land raw messages before deserializing them. The principle is the same; the implementation details differ because
One of the recurring costs in a Kafka-based pipeline is writing the glue code. You have a source — a database, an API, a file drop — and you need to get its data into a topic. Writing a producer from scratch means handling connection management, offset tracking, error recovery, and
Running Spark on EMR yourself sounds like freedom. You pick the instance types, you tune the Spark config, you control the versions, you pay only for what you use. What the pitch leaves out: all of that is also your problem at 2 a.m. when a job fails due
If you are doing data engineering in an Azure shop and you have been eyeing Kafka, Microsoft has an answer for you: Azure Event Hubs. It shipped into general availability earlier this year and it is worth understanding both what it is and where it differs from the open-source
A pipeline failed at 3 AM. The client calls at 9 AM asking why their morning report is empty. I open the ADF monitor. I see a red slice. The error message says "SqlException: Timeout expired." That's it. No row counts, no which activity failed first,
The previous post was about the geocoding job that produced zero rows. This post is about the broader problem it revealed: ADLA's monitoring story in 2015 was inadequate for detecting jobs that ran correctly but did the wrong thing. Here's what ADLA showed you, what it
When I first encountered Biml, my honest reaction was something like: "This is just an XML version of my _Templates folder, except it generates the packages instead of making me copy them." That reaction turned out to be entirely correct, and understanding it that way made Biml immediately
Teams migrating from on-prem Hadoop to AWS EMR in 2015 are making a predictable mistake: they're treating S3 like a remote HDFS. Same design patterns, same small-file structure, same assumptions about read performance. The migration works — the jobs run — but the costs are higher than expected
I need to tell you about the geocoding job. This is the story of a U-SQL job that ran all its vertices, consumed 200 Analytics Units for six hours, reported 100% completion, and produced a output file with zero rows. Not an error. Not a failure status. Success — and
ADF has been generally available for months. The connector list has grown. Copy Activity has matured. PolyBase support for SQL Data Warehouse is excellent. And there is still no git integration. I've been patient about this. I'm done being patient. What Git Integration Would Actually Mean