Two years ago I started running Azure Data Factory in production for a client that needed to move data from a half-dozen on-premises SQL Server instances into Azure SQL Data Warehouse. The honest sales pitch at the time was: managed cloud ETL, no servers to maintain, connector library that would save us from writing custom integration code. Two years of production use is enough time to separate what the pitch promised from what the platform actually delivered.
Time to take stock.
What Actually Held Up
Copy Activity Reliability
The Copy Activity is the core of ADF and it works. Not glamorously, not without configuration — but once you've tuned the retry policies and understood the slice model, it runs. We've had pipelines moving multi-gigabyte files daily for eighteen months without intervention. The retry logic handles transient failures cleanly: source throttling, momentary network hiccups, Azure SQL connection pool exhaustion. Set retryCount to 3 and retryInterval to 00:05:00 and you stop getting paged at 2am for things that aren't actually problems.
Data Management Gateway Stability
The DMG — the on-premises agent that gives ADF connectivity to SQL Server instances inside your network — has been more stable than I expected. We're running it on a pair of Windows Server VMs for redundancy. After an initial round of certificate troubleshooting that I'd rather forget, it's been reliable. Upgrades are manual and slightly nerve-wracking (you take the agent offline briefly), but they've gone smoothly. For the primary use case — SQL Server to Azure — it does the job.
The JSON Deployment Model
ADF v1 expresses everything as JSON: pipelines, datasets, linked services, tables. Once you accept that the portal is a lie and the JSON files are the truth, you can build a reasonable deployment workflow. We export every pipeline and dataset definition into a git repository after any change. Deployments go through PowerShell:
New-AzureRmDataFactoryPipeline -ResourceGroupName $rg `
-DataFactoryName $adf `
-Name $pipelineName `
-File ".\pipelines\$pipelineName.json" `
-ForceIt's not pretty. It's not what I'd design if starting fresh. But it works, and it's auditable. We know exactly what's deployed because we deployed it from source.
Connector Library Growth
The connector library in 2016 is legitimately good. Azure Blob, Azure Data Lake Store, Azure SQL, Azure SQL Data Warehouse via PolyBase, SQL Server on-premises via DMG, Oracle via DMG, MySQL, FTP, SFTP, Salesforce. For most enterprise integration scenarios, you're not writing custom code. That was not true in 2014. This is real progress.
What Didn't Hold Up
The Parameterization Gap Is Getting Worse
The absence of native parameterization in ADF v1 was annoying in 2014. In 2016, with sixty-plus pipeline JSON files in our repository, it's a maintenance crisis in slow motion. Every table we ingest has its own pipeline definition. They're structurally identical — same retry settings, same error handling, same sink configuration — but because you can't parameterize a pipeline in v1, you have a separate JSON file per table.
The metadata-driven framework I tried to build in 2015 is still running, but the seams show. It works around the parameterization gap using stored procedures and dynamic SQL to loop over a config table. It's clever. It's also fragile, hard to explain to new team members, and depends on behaviors that aren't documented. Not ideal.
Monitoring Is Still Thin
The ADF portal monitoring view shows you slice status: Ready, In Progress, Failed, Waiting. That's it. No row counts mid-run, no throughput graphs, no timing breakdown between data read time and write time. When a pipeline is slow, you're guessing at the cause. Is it the source query? Network saturation? The sink writing slowly? ADF won't tell you.
We've built our own monitoring layer: a stored procedure activity at the end of each pipeline writes status and timing to an Azure SQL logging table. It works but it's custom code that every team member has to understand. This should be a platform feature.
The Git Gap Compounds
Here's the thing about not having native git integration: the longer you run without it, the more damage the gap causes. We started with 10 pipelines. We have 60 now. The operational discipline required to keep the portal and git in sync is real work. It doesn't scale as the team grows. I've had teammates edit pipelines directly in the portal to fix production issues — understandable when something is broken at midnight — and then not export the JSON afterward. We've caught those drifts during deployment. We haven't lost work yet, but we've come close.
Outlook for 2016
Microsoft previewed ADF v2 late last year and the architecture is substantially different. Parameterization is built in. The scheduling model is redesigned. The connector story continues to improve. The migration from v1 to v2 is a rebuild, not an upgrade — the JSON schema is different enough that your existing pipelines don't port directly. That's a real cost.
My plan for 2016: run v1 in production, watch v2 in preview, start building v2 equivalents for new workloads. The v2 parameterization feature alone is worth the migration effort. I'll write more about v2 as I get hands-on time with the preview.
If you're just starting with ADF, start with v2. If you're running v1 in production and it's working, don't migrate until v2 hits GA. And if you're maintaining 60 pipeline JSON files with manual git discipline, you have my sympathy. I'm here to help.