I've been doing SSIS development since SQL Server 2005. I know where every body is buried: the Data Flow's memory pressure at scale, the deployment model improvements in the 2012 project model, the debugger with data viewer that saves hours of "what is actually in
Kafka's retention model is one of the properties that separates it from a traditional message queue, and it is also one of the properties that most often surprises people the first time their consumer goes down for a few days. Here is the scenario: your raw landing consumer
Somewhere along the way, "streaming" became synonymous with "always-on, real-time, millisecond latency." That framing does a lot of damage. It makes people think they have to choose: either run a consumer 24 hours a day, 7 days a week, paying for compute continuously, or
When you come to ADF from SSIS, you carry a mental model of packages, connection managers, data flows, and control flows. Some of that maps cleanly to ADF. Some of it maps badly enough to cause real confusion. Let's build the right mental model from the start. Linked
The conceptual pieces make sense — topics, partitions, offsets, consumer groups. Now you need to actually run something. This post walks through a minimal but realistic producer and consumer using the kafka-python library. I am assuming you have a Kafka broker running somewhere (local or remote) and Python 2.7
The upstream team added a column to their events table. Routine change, they said. They updated the Hive metastore, ran their own jobs to verify the new column populated correctly, and called it done. What they didn't do was tell anyone downstream. Six hours later, three downstream jobs
After inheriting the 200-package SSIS environment, the solution was clear: generate the packages programmatically from a metadata store rather than writing them by hand. The resulting system was one of the highest-leverage things I built during this period — adding a new source to the warehouse went from a
Last year I covered running tSQLt in a build pipeline via sqlcmd. That works, but it has a coupling problem: the tests live in the database, not in source control. If the database is recreated from scratch, the tests are gone. If someone deletes a test procedure directly, it'
Every few years a tool shows up that makes you stop and reconsider your defaults. For most of us doing ETL on the Microsoft stack, SSIS is that default. It works, it's mature, you know exactly where to look when it breaks. Then Azure Data Factory hit preview
Nobody tested their stored procedures in 2010. I mean that almost literally — the pattern was: write the procedure, run it manually with a few sample inputs, see if it does approximately what you wanted. Automated tests for database logic were theoretically possible but practically nonexistent in most shops. SSDT and
Every data project I have ever been on started with someone telling me what they needed the system to do. Not how — what. "We need to load customer orders from the source system into the warehouse by 6 a.m. every day." "We need to flag records
A dacpac contains the schema model. A publish profile controls how that model gets applied to a specific environment. Without publish profiles, every deployment would use the same settings — the same permissiveness for destructive changes, the same connection details, the same behavior for out-of-sync objects. With publish profiles,