Azure Data Factory has been generally available for over a year. It still has no git integration. I want to be precise about what that means, because "no git integration" sounds like a developer convenience feature. It isn't. It's a production operations problem. What
Kafka 0.9.0 shipped in November 2015 with a rewritten consumer API, and it changes the conversation about where event streaming fits in a data pipeline architecture. The old consumer API was a Zookeeper-coupled, thread-unsafe mess that made Kafka feel like infrastructure you tolerated rather than infrastructure
Your IoT devices speak MQTT. Your analytics pipeline speaks Kafka. The bridge between them is not complicated, but there are enough configuration choices that it is worth walking through a concrete setup. The goal: every message published to your MQTT broker gets written as a Kafka message, with the MQTT
Here is a failure mode I have seen enough times to write a post about it. A producer team updates their message schema — adds a field, renames one, changes a type from string to integer. They tell nobody. The consumer deserialization code breaks. Or worse, it does not break — it
The benchmark numbers are in the previous post. This one explains the mechanism — specifically why a C# Parallel.ForEach beat SQL Server's MAXDOP parallelism for geocoding, and how to think about when that outcome should make you reach for a C# parallel program instead of a SQL query.
Two years ago I started running Azure Data Factory in production for a client that needed to move data from a half-dozen on-premises SQL Server instances into Azure SQL Data Warehouse. The honest sales pitch at the time was: managed cloud ETL, no servers to maintain, connector library
If your streaming processing requirements are straightforward — filtering, aggregating, joining two topics — you probably do not need a separate compute cluster. Kafka Streams, introduced in Kafka 0.10, is a Java library that runs stream processing inside your application process. No YARN. No separate cluster. No Zookeeper dependency beyond what
I want to give you a specific benchmark. Not vague "significantly faster" marketing language — an actual result from a real workload that I measured, on identical hardware, and can explain mechanically. Here it is: geocoding 23 years of NOAA storm event radar data, C# with Parallel.ForEach beat
A year of ADF in production. Time to give you the actual decision framework — not "it depends," but specific criteria that determine which tool belongs on which workload. I've used both in real client engagements in 2015. Here's what I know. Start With Where
Here is the most common anti-pattern I see in data pipeline code: the pipeline knows the names of its tables. It knows the column names it's going to transform. It knows the business rules. It knows the output schema. All of this — structural knowledge and business logic
I wrote my first tSQLt test in late 2012. Three years later, I've used it across multiple projects, introduced it to teams that had never tested SQL before, and hit most of the sharp edges the framework has. It's time for an honest assessment. What tSQLt
By mid-2015, the Azure analytical data platform had two serious options: Azure Data Lake Analytics with U-SQL, and Azure SQL Data Warehouse (Azure DW). I was using both in different client contexts and kept getting asked which one to use for a given problem. The answer wasn'