There is a folder on my laptop — and its predecessors, going back farther than I want to admit — called _Templates. Inside it: a master SSIS package called MASTER_ControlFlow.dtsx. A SQL file called CREATE_TABLE_standard.sql. A file called usp_standard_skeleton.sql. A README.txt that explains
I've been doing SSIS development since SQL Server 2005. I know where every body is buried: the Data Flow's memory pressure at scale, the deployment model improvements in the 2012 project model, the debugger with data viewer that saves hours of "what is actually in
The previous post covered the architecture: metadata store design, generator structure, template approach. This one gets into the implementation details — specifically, how the Microsoft.SqlServer.ManagedDTS API works for programmatically building data flow tasks, and the gotchas I hit along the way. Setting Up the Data Flow Task // Add a
Kafka's retention model is one of the properties that separates it from a traditional message queue, and it is also one of the properties that most often surprises people the first time their consumer goes down for a few days. Here is the scenario: your raw landing consumer
Somewhere along the way, "streaming" became synonymous with "always-on, real-time, millisecond latency." That framing does a lot of damage. It makes people think they have to choose: either run a consumer 24 hours a day, 7 days a week, paying for compute continuously, or stick with
The conceptual pieces make sense — topics, partitions, offsets, consumer groups. Now you need to actually run something. This post walks through a minimal but realistic producer and consumer using the kafka-python library. I am assuming you have a Kafka broker running somewhere (local or remote) and Python 2.7 or
When you come to ADF from SSIS, you carry a mental model of packages, connection managers, data flows, and control flows. Some of that maps cleanly to ADF. Some of it maps badly enough to cause real confusion. Let's build the right mental model from the start. Linked
The upstream team added a column to their events table. Routine change, they said. They updated the Hive metastore, ran their own jobs to verify the new column populated correctly, and called it done. What they didn't do was tell anyone downstream. Six hours later, three downstream jobs
After inheriting the 200-package SSIS environment, the solution was clear: generate the packages programmatically from a metadata store rather than writing them by hand. The resulting system was one of the highest-leverage things I built during this period — adding a new source to the warehouse went from a half-day manual
Last year I covered running tSQLt in a build pipeline via sqlcmd. That works, but it has a coupling problem: the tests live in the database, not in source control. If the database is recreated from scratch, the tests are gone. If someone deletes a test procedure directly, it'
Every few years a tool shows up that makes you stop and reconsider your defaults. For most of us doing ETL on the Microsoft stack, SSIS is that default. It works, it's mature, you know exactly where to look when it breaks. Then Azure Data Factory hit preview
Every data project I have ever been on started with someone telling me what they needed the system to do. Not how — what. "We need to load customer orders from the source system into the warehouse by 6 a.m. every day." "We need to flag records