A client engagement in Q3 2014 pushed me to put ADF into actual production — not a proof of concept, not a sandbox, but pipelines moving real data that real people depend on for real decisions. Six months later, I have an honest report on what held up and what didn&
The pitch for Spark in early 2015 is always the same: it's faster than MapReduce. The benchmark slides come out. Spark wins by 10x, 100x, sometimes more. The audience nods. The Hadoop cluster procurement gets delayed. Speed is real. But if speed is the only reason you'
It's December 2014. ADF has been in preview most of the year. I've built pipelines with it, watched it fail in ways SSIS doesn't, and watched it succeed in ways SSIS can't. Time for an honest year-end assessment before the hype either
U-SQL is Microsoft's answer to a question Hive and Pig were already answering for the Hadoop ecosystem: how do you write analytical queries over big data that non-MapReduce programmers can understand? The Hive answer was "SQL-ish syntax that compiles to MapReduce." The Pig answer was "
I spend a fair amount of time in these posts pointing out what ADF is missing. That's appropriate — gaps matter when you're deciding whether to use a tool in production. But ADF also makes decisions that SSIS was architecturally never going to make, and those decisions
Before Azure Data Lake Storage, storing large amounts of unstructured data in Azure meant Azure Blob Storage. Blob Storage worked, but it was an object store with a flat namespace — "folders" were really just prefixes in the object name. If you wanted to use it as the foundation
Copy Activity is ADF's engine. It is the one activity you will use in nearly every pipeline, and it has more knobs than the documentation makes obvious. I've spent enough time with it now to give you the full picture — what it can do, how to
By late 2014, I had built metadata-driven generators for SSIS packages, Azure Data Factory pipelines, and the early pieces of a metadata-driven approach to SSDT publication profiles. The same pattern kept emerging in different contexts, and at some point it was worth stepping back and articulating why the pattern worked
Getting tSQLt into TeamCity is where individual tests become a team safety net. Instead of one developer running tests in SSMS occasionally, every commit triggers a build that deploys the schema and runs the full suite. Broken tests block the merge. Passing tests give the whole team confidence to ship.
The metadata-driven generation pattern I built for SSIS packages applied almost directly to ADF. The metadata store already described source tables, column mappings, and destination tables. ADF pipelines were JSON. The generator just needed to target a different output format. Here's what was different about generating for ADF
Every data store your ADF pipeline touches — source or sink — starts with a linked service. Get the linked service right and the rest of the pipeline follows. Get it wrong and you spend an afternoon staring at authentication errors. Let's cover what's available in 2014 and
Azure Data Factory version 1 launched in preview in 2014 and I started using it almost immediately. The pitch was familiar: a pipeline orchestration service for moving and transforming data, running as a managed Azure service rather than a server you operated. For someone who'd spent two years