I've been using Microsoft data tools professionally since SQL Server 2000. SSIS since 2005. Azure Data Factory since 2014. That's long enough to recognize a pattern in how Microsoft ships products, and long enough to have an opinion about what that pattern costs the people who use those products.
The pattern: Microsoft ships when the product covers 80% of use cases adequately. The remaining 20% — the edge cases, the enterprise requirements, the operational tooling that serious production users need — gets shipped later, sometimes much later, sometimes as a third-party ecosystem, sometimes never.
I've started calling this the 80% rule. Let me tell you what it means for ADF users specifically, and what I think you should do about it.
The Historical Evidence
SSIS shipped in 2005 without a metadata-driven framework. SQL Server had been in production for years. Every serious data warehouse shop had a need for generic, configurable ETL patterns — load any table using a config-driven approach, without hand-writing a package per table. SSIS shipped without built-in support for this. The community built BIML (BI Markup Language) to generate SSIS packages programmatically. BIML wasn't Microsoft's answer. BIML was the community's answer to Microsoft's absence.
SSIS also shipped without a clean CI/CD story. Package deployment was manual until SQL Server 2012 introduced the Project Deployment Model with SSISDB. Seven years. The community ran various deployment scripts, source control workarounds, and build automation hacks for seven years before Microsoft addressed it properly.
ADF followed the same pattern. Shipped in 2014 without git integration. Shipped without parameterization. Shipped without a looping construct. Three years later, parameterization and ForEach arrived. Git integration — the most-requested feature by vote count on the Azure feedback portal for most of ADF's existence — is still not there at the end of 2017.
This is not coincidence. This is a product management approach.
Why Microsoft Does This
I want to be precise here because "Microsoft is lazy" is the easy answer and it's wrong.
Shipping the 80% gets the product to market. Market presence generates customer signal. Customer signal drives roadmap prioritization. The features that matter most to the largest number of customers get built first. The features that matter most to the most sophisticated users — git integration, advanced CI/CD, enterprise monitoring — come later because the signal for them comes from a smaller, louder segment of the customer base.
This is rational product management if your goal is adoption. It's not rational if your goal is avoiding technical debt for early adopters.
Microsoft also has a history of relying on the community to build the last 20%. BIML is a real product now — Varigence built a company around it. The ADF community has built export-to-git tools, deployment scripts, monitoring patterns. These community solutions work. But they come with a cost: fragmentation, support uncertainty, and dependency on third parties for functionality that should be in the platform.
What the 80% Rule Costs ADF Users
Let me count the specific costs I've documented across three years of production ADF:
The git gap: every serious ADF shop independently invented their own export-to-git workflow. Most of them have had at least one incident where a portal change wasn't exported and was subsequently overwritten by a deployment. The cost of that incident: investigation time, potential data gap, team trust erosion. Multiply by every ADF shop that hit this before native git integration ships.
The parameterization gap: every ADF shop that needed generic pipelines before v2 invented a workaround. Stored procedure loops, external API calls, duplicated JSON files. Some of those workarounds are now technical debt in production systems that teams will spend months removing after migrating to v2 parameterization.
The ForEach gap: the pattern arrived in October 2017 preview. Teams that needed metadata-driven frameworks before that date either built elaborate workarounds or chose a different tool. Some chose the different tool and aren't coming back to ADF.
The monitoring gap: every production ADF implementation has a custom monitoring solution. Azure Monitor alert rules, custom audit log tables, Power BI dashboards. All independently invented. All maintained in addition to the actual data pipelines.
What You Should Do About It
Accept the pattern. It's not going to change. Microsoft has been shipping at 80% for twenty years and it works well enough for them commercially that there's no incentive to change the approach.
Build the 20% yourself and document it explicitly. The export-to-git workflow, the monitoring pattern, the audit log table, the metadata-driven framework — these are your technical standards, and they belong in your documentation and your code review requirements. Don't hope Microsoft will ship them before you need them.
Track the roadmap actively. The 20% does eventually ship. When it ships, your workaround becomes technical debt. If you know what's coming and roughly when, you can plan the migration from workaround to native feature. The Azure roadmap, the product team's GitHub issues, the feedback portal vote counts — these are signal. Read them.
Don't confuse "this is missing" with "this tool is wrong." ADF in 2017 is substantially better than ADF in 2014. The trajectory has been consistently positive. The gaps are real and they have real costs, but the core platform has proven reliable at scale and the connector library is genuinely comprehensive. The tool is right for most cloud data integration work. It just ships incomplete and asks you to finish it.
Looking Forward
Git integration is the outstanding feature at the end of 2017. I expect it to ship in 2018. When it does, the export-to-git discipline I've been preaching since 2014 becomes optional. The discipline you built won't hurt you — git is still the right source control, and native integration just removes the manual ceremony. But the discipline cost goes away.
The ADF v2 GA announcement is expected soon. With ForEach, parameterization, three IR types, three trigger types, and the connector library at 90+, the platform is feature-complete for the scenarios that cover 80% of production workloads. The remaining 20% — git, better monitoring, cross-pipeline dependencies — will arrive on Microsoft's schedule, not yours.
Plan accordingly. Build the workarounds you need. Track the roadmap. Migrate to native features when they arrive. It's not the product experience I'd design, but it's the one we have, and it's workable. As always, I'm here to help.