ADF has been generally available for months. The connector list has grown. Copy Activity has matured. PolyBase support for SQL Data Warehouse is excellent. And there is still no git integration. I've been patient about this. I'm done being patient. What Git Integration Would Actually Mean
I need to tell you about the geocoding job. This is the story of a U-SQL job that ran all its vertices, consumed 200 Analytics Units for six hours, reported 100% completion, and produced a output file with zero rows. Not an error. Not a failure status. Success — and nothing
FakeTable handles table dependencies. SpyProcedure handles stored procedure dependencies. The gap that remained for a while was scalar functions: if your stored procedure called dbo.GetCurrentExchangeRate() or dbo.IsBusinessDay(@Date), you had no clean way to control what those functions returned during a test. The results depended on whatever was
Copy Activity has matured since the 2014 preview. New source and sink types, PolyBase support for SQL Data Warehouse, and enough production experience to know the patterns that work and the ones that fight you. This is the deep dive SSIS developers need before they build their first production ADF
The data pipeline starts at 2 a.m. It's a cron job. It reads from a raw events table, transforms it, and writes to a reporting table. The reporting table is what the business sees at 8 a.m. The upstream job that populates the raw events table
The connector list has grown significantly since ADF's 2014 preview. Before I cover the new connectors, I want to go deep on datasets — specifically the partition model and the external flag, which are the two most common sources of pipeline failures for developers new to ADF. Updated Connector
The U-SQL introduction post was conceptual. This one is practical: given raw CSV files in ADLS, here's how to go from nothing to a clean, queryable dataset in Azure Data Lake Analytics. I'll use the NOAA storm events data I was working with at the time
A year ago I wrote the initial mapping of ADF concepts to SSIS equivalents. After a year of production experience, I know which mappings hold and which ones break down badly enough to cause real problems. Let's revisit this with calibrated expectations. Mappings That Hold Linked Service = Connection
By 2015, continuous integration was standard for application code: every commit triggered a build, every build ran tests, the green/red status was visible. The database was frequently the exception: schema changes deployed manually by a DBA, tested in staging only when someone remembered to, deployed to production during a
A tSQLt suite with 10 tests is easy to manage. A suite with 200 tests is a different problem. The test class structure that felt obvious when you started starts to break down: tests that belong together aren't, test names that made sense in isolation are ambiguous at
A client engagement in Q3 2014 pushed me to put ADF into actual production — not a proof of concept, not a sandbox, but pipelines moving real data that real people depend on for real decisions. Six months later, I have an honest report on what held up and what didn&
The pitch for Spark in early 2015 is always the same: it's faster than MapReduce. The benchmark slides come out. Spark wins by 10x, 100x, sometimes more. The audience nods. The Hadoop cluster procurement gets delayed. Speed is real. But if speed is the only reason you'