I wrote my first tSQLt test in late 2012. Three years later, I've used it across multiple projects, introduced it to teams that had never tested SQL before, and hit most of the sharp edges the framework has. It's time for an honest assessment. What tSQLt
By mid-2015, the Azure analytical data platform had two serious options: Azure Data Lake Analytics with U-SQL, and Azure SQL Data Warehouse (Azure DW). I was using both in different client contexts and kept getting asked which one to use for a given problem. The answer wasn't "
If you've done SSIS development at scale, you know BIML. You define a config table — source tables, target tables, column mappings, load patterns — and BIML generates N perfectly isolated SSIS packages at compile time. Each package is its own executable, its own set of connection managers, its own
Airflow open-sourced out of Airbnb in June 2015. I've been running it internally for a few weeks now and I want to explain the core idea, because it's easy to look at the setup overhead and dismiss it as "Luigi with a web UI."
The three-stage streaming pipeline is not a new idea — it is the streaming equivalent of a well-understood batch ETL pattern. In batch, you stage raw files before loading them. In streaming, you land raw messages before deserializing them. The principle is the same; the implementation details differ because the data
One of the recurring costs in a Kafka-based pipeline is writing the glue code. You have a source — a database, an API, a file drop — and you need to get its data into a topic. Writing a producer from scratch means handling connection management, offset tracking, error recovery, and schema
If you are doing data engineering in an Azure shop and you have been eyeing Kafka, Microsoft has an answer for you: Azure Event Hubs. It shipped into general availability earlier this year and it is worth understanding both what it is and where it differs from the open-source alternative.
Running Spark on EMR yourself sounds like freedom. You pick the instance types, you tune the Spark config, you control the versions, you pay only for what you use. What the pitch leaves out: all of that is also your problem at 2 a.m. when a job fails due
A pipeline failed at 3 AM. The client calls at 9 AM asking why their morning report is empty. I open the ADF monitor. I see a red slice. The error message says "SqlException: Timeout expired." That's it. No row counts, no which activity failed first,
The previous post was about the geocoding job that produced zero rows. This post is about the broader problem it revealed: ADLA's monitoring story in 2015 was inadequate for detecting jobs that ran correctly but did the wrong thing. Here's what ADLA showed you, what it
Teams migrating from on-prem Hadoop to AWS EMR in 2015 are making a predictable mistake: they're treating S3 like a remote HDFS. Same design patterns, same small-file structure, same assumptions about read performance. The migration works — the jobs run — but the costs are higher than expected and the
When I first encountered Biml, my honest reaction was something like: "This is just an XML version of my _Templates folder, except it generates the packages instead of making me copy them." That reaction turned out to be entirely correct, and understanding it that way made Biml immediately