Teams migrating from on-prem Hadoop to AWS EMR in 2015 are making a predictable mistake: they're treating S3 like a remote HDFS. Same design patterns, same small-file structure, same assumptions about read performance. The migration works — the jobs run — but the costs are higher than expected and the
When I first encountered Biml, my honest reaction was something like: "This is just an XML version of my _Templates folder, except it generates the packages instead of making me copy them." That reaction turned out to be entirely correct, and understanding it that way made Biml immediately
ADF has been generally available for months. The connector list has grown. Copy Activity has matured. PolyBase support for SQL Data Warehouse is excellent. And there is still no git integration. I've been patient about this. I'm done being patient. What Git Integration Would Actually Mean
I need to tell you about the geocoding job. This is the story of a U-SQL job that ran all its vertices, consumed 200 Analytics Units for six hours, reported 100% completion, and produced a output file with zero rows. Not an error. Not a failure status. Success — and nothing
FakeTable handles table dependencies. SpyProcedure handles stored procedure dependencies. The gap that remained for a while was scalar functions: if your stored procedure called dbo.GetCurrentExchangeRate() or dbo.IsBusinessDay(@Date), you had no clean way to control what those functions returned during a test. The results depended on whatever was
Copy Activity has matured since the 2014 preview. New source and sink types, PolyBase support for SQL Data Warehouse, and enough production experience to know the patterns that work and the ones that fight you. This is the deep dive SSIS developers need before they build their first production ADF
The data pipeline starts at 2 a.m. It's a cron job. It reads from a raw events table, transforms it, and writes to a reporting table. The reporting table is what the business sees at 8 a.m. The upstream job that populates the raw events table
The connector list has grown significantly since ADF's 2014 preview. Before I cover the new connectors, I want to go deep on datasets — specifically the partition model and the external flag, which are the two most common sources of pipeline failures for developers new to ADF. Updated Connector
The U-SQL introduction post was conceptual. This one is practical: given raw CSV files in ADLS, here's how to go from nothing to a clean, queryable dataset in Azure Data Lake Analytics. I'll use the NOAA storm events data I was working with at the time
A year ago I wrote the initial mapping of ADF concepts to SSIS equivalents. After a year of production experience, I know which mappings hold and which ones break down badly enough to cause real problems. Let's revisit this with calibrated expectations. Mappings That Hold Linked Service = Connection
By 2015, continuous integration was standard for application code: every commit triggered a build, every build ran tests, the green/red status was visible. The database was frequently the exception: schema changes deployed manually by a DBA, tested in staging only when someone remembered to, deployed to production during a
A tSQLt suite with 10 tests is easy to manage. A suite with 200 tests is a different problem. The test class structure that felt obvious when you started starts to break down: tests that belong together aren't, test names that made sense in isolation are ambiguous at