The number one question I get from engineers new to ADF is some variation of: "I set up the pipeline, it says it's running, but no data moved. What's happening?" Nine times out of ten, the answer involves the slice model. Nine times out
I want to talk about Snowflake — not to dismiss it, because it genuinely solves real problems, but to be precise about the nature of the relationship you're entering when you build on it. "Convenience" and "dependency" are not the same word, but in practice
Docker went 1.0 in 2014, but it took most of the enterprise data engineering world until 2016 to start paying serious attention. By that point, application developers had been containerizing web services for two years, and the lessons were well enough established that it was worth applying them to
SQL Server 2016 RTM shipped last year and most teams are moving to it now. If you have a tSQLt suite — and by this point you should — there are specific things worth verifying before the upgrade touches your test databases. None of them are dealbreakers, but skipping the checks means
Azure Data Factory has been generally available for over a year. It still has no git integration. I want to be precise about what that means, because "no git integration" sounds like a developer convenience feature. It isn't. It's a production operations problem. What
Kafka 0.9.0 shipped in November 2015 with a rewritten consumer API, and it changes the conversation about where event streaming fits in a data pipeline architecture. The old consumer API was a Zookeeper-coupled, thread-unsafe mess that made Kafka feel like infrastructure you tolerated rather than infrastructure you built
Your IoT devices speak MQTT. Your analytics pipeline speaks Kafka. The bridge between them is not complicated, but there are enough configuration choices that it is worth walking through a concrete setup. The goal: every message published to your MQTT broker gets written as a Kafka message, with the MQTT
Here is a failure mode I have seen enough times to write a post about it. A producer team updates their message schema — adds a field, renames one, changes a type from string to integer. They tell nobody. The consumer deserialization code breaks. Or worse, it does not break — it
If your streaming processing requirements are straightforward — filtering, aggregating, joining two topics — you probably do not need a separate compute cluster. Kafka Streams, introduced in Kafka 0.10, is a Java library that runs stream processing inside your application process. No YARN. No separate cluster. No Zookeeper dependency beyond what
Two years ago I started running Azure Data Factory in production for a client that needed to move data from a half-dozen on-premises SQL Server instances into Azure SQL Data Warehouse. The honest sales pitch at the time was: managed cloud ETL, no servers to maintain, connector library that would
The benchmark numbers are in the previous post. This one explains the mechanism — specifically why a C# Parallel.ForEach beat SQL Server's MAXDOP parallelism for geocoding, and how to think about when that outcome should make you reach for a C# parallel program instead of a SQL query.
A year of ADF in production. Time to give you the actual decision framework — not "it depends," but specific criteria that determine which tool belongs on which workload. I've used both in real client engagements in 2015. Here's what I know. Start With Where