I spend a fair amount of time in these posts pointing out what ADF is missing. That's appropriate — gaps matter when you're deciding whether to use a tool in production. But ADF also makes decisions that SSIS was architecturally never going to make, and those decisions
Before Azure Data Lake Storage, storing large amounts of unstructured data in Azure meant Azure Blob Storage. Blob Storage worked, but it was an object store with a flat namespace — "folders" were really just prefixes in the object name. If you wanted to use it as the foundation
Copy Activity is ADF's engine. It is the one activity you will use in nearly every pipeline, and it has more knobs than the documentation makes obvious. I've spent enough time with it now to give you the full picture — what it can do, how to
By late 2014, I had built metadata-driven generators for SSIS packages, Azure Data Factory pipelines, and the early pieces of a metadata-driven approach to SSDT publication profiles. The same pattern kept emerging in different contexts, and at some point it was worth stepping back and articulating why the
Getting tSQLt into TeamCity is where individual tests become a team safety net. Instead of one developer running tests in SSMS occasionally, every commit triggers a build that deploys the schema and runs the full suite. Broken tests block the merge. Passing tests give the whole team confidence to ship.
The metadata-driven generation pattern I built for SSIS packages applied almost directly to ADF. The metadata store already described source tables, column mappings, and destination tables. ADF pipelines were JSON. The generator just needed to target a different output format. Here's what was different about generating for
Azure Data Factory version 1 launched in preview in 2014 and I started using it almost immediately. The pitch was familiar: a pipeline orchestration service for moving and transforming data, running as a managed Azure service rather than a server you operated. For someone who'd spent two years
Every data store your ADF pipeline touches — source or sink — starts with a linked service. Get the linked service right and the rest of the pipeline follows. Get it wrong and you spend an afternoon staring at authentication errors. Let's cover what's available in 2014 and
Most tSQLt tutorials start with stored procedures, and for good reason — that's where most of the testable business logic lives. But triggers, views, and scalar functions all contain logic worth testing, and tSQLt handles all of them with the same core primitives. Testing Triggers Triggers are the hardest
The ADF portal makes setup look deceptively simple. Create a data factory, click around, start building pipelines. What it doesn't tell you upfront: there are prerequisites, gotchas with on-premises connectivity, and a browser-based editor that will cost you work if you don't build a
There is a folder on my laptop — and its predecessors, going back farther than I want to admit — called _Templates. Inside it: a master SSIS package called MASTER_ControlFlow.dtsx. A SQL file called CREATE_TABLE_standard.sql. A file called usp_standard_skeleton.sql. A README.txt that explains
The previous post covered the architecture: metadata store design, generator structure, template approach. This one gets into the implementation details — specifically, how the Microsoft.SqlServer.ManagedDTS API works for programmatically building data flow tasks, and the gotchas I hit along the way. Setting Up the Data Flow Task // Add a