The Question I've Been Asking for Ten Years
Every time Microsoft ships a new version of its data integration platform, I ask the same question: has the metadata-driven framework gap finally closed? The pattern — control table, ForEach loop, parameterized pipeline template, one pipeline that handles N tables — has been the standard approach for scalable ADF implementation since v2. And Microsoft has never shipped first-party support for it.
I checked Fabric Data Factory carefully. The answer is no. Again.
Let me explain what that means, why I expected it, and what to do about it.
What "Metadata-Driven" Means
For anyone newer to ADF who hasn't lived through the evolution: the metadata-driven pattern solves the "I have 200 tables to load" problem without creating 200 pipelines.
The pattern:
- A control table (in Azure SQL, or any queryable store) holds one row per table: source connection, source table name, destination table name, watermark column, load type, active flag, and any other parameters your pipeline needs.
- A parent pipeline does a Lookup activity against the control table (SELECT * WHERE active = 1). The result is a list of rows.
- A ForEach activity iterates over the Lookup output, calling a child pipeline once per row.
- The child pipeline is fully parameterized — it accepts all the per-table values as parameters and runs the Copy Activity (or Data Flow) with those parameters.
Result: 200 tables, 2 pipelines (parent and child), and a control table. Add a new table: insert a row. Change load behavior for a table: update a row. Disable a table temporarily: set active = 0. No pipeline authoring required.
This pattern is the right pattern. It's been the right pattern since ADF v2 shipped parameterization. Microsoft has known about it for years — community blog posts, conference presentations, GitHub samples. They've never made it first-party.
What Fabric Data Factory Provides
The tools needed to implement the metadata-driven pattern exist in Fabric Data Factory exactly as they existed in standalone ADF:
- Lookup activity: same behavior, same configuration, same output format
- ForEach activity: same iteration model, same parallelism controls (sequential or parallel with concurrency limit)
- Execute Pipeline activity: same parameter passing mechanism
- Pipeline parameters: same parameterization model at pipeline, dataset, and linked service level
If you have a metadata-driven framework in ADF, it runs in Fabric Data Factory. No changes needed to the pattern. The control table, the parent pipeline, the child pipeline, the parameter passing — all of it transfers.
What Fabric Data Factory does not provide: a first-party template, wizard, documentation, or tooling for creating this pattern. You still build it yourself.
Why I Expected This
Microsoft's approach to orchestration tooling has been consistent across SSIS, ADF, and now Fabric: they ship the engine at 80-90% completeness and rely on the community to build the framework layer.
SSIS shipped without a first-party metadata-driven framework. BIML emerged from the community to fill that gap. BIML generated SSIS packages programmatically from metadata — the exact pattern SSIS didn't provide natively.
ADF shipped without a first-party metadata-driven framework. The community built it: blog series, GitHub repos, conference talks, consulting engagements. Every ADF shop that needed to load more than 20 tables built their own version of this pattern.
Fabric is shipping without a first-party metadata-driven framework. The community will build it again. The tooling will be different (Fabric workspaces, OneLake, the new git model), but the pattern will be the same.
This is not a failure of imagination at Microsoft. It may be a deliberate product decision — keep the engine general-purpose, let the ecosystem build the frameworks. That decision has been good for Microsoft (frameworks create ecosystem lock-in, which is stickier than product features) and mixed for customers (you own the framework you build, but you also own the maintenance burden).
What This Means for Your Fabric Migration
If you've already built a metadata-driven framework in ADF, your migration to Fabric Data Factory is straightforward for this pattern:
- Create the equivalent pipelines in Fabric
- Point the Lookup activity at your existing control table (linked service to Azure SQL or wherever your control table lives)
- The ForEach → Execute Pipeline → parameterized child pipeline pattern works identically
Your control table schema doesn't need to change. Your parameter conventions don't need to change. The pattern is portable.
The one operational difference: the control table linked service configuration in Fabric uses Fabric's linked service model. If your control table is in an Azure SQL database, the linked service setup is the same as ADF; the connection configuration looks identical.
The Ten-Year Pattern
I've been writing about the metadata-driven gap since 2015. SSIS didn't close it. ADF v1 didn't close it. ADF v2 didn't close it. Fabric Data Factory doesn't close it.
I'm done expecting Microsoft to close it. Build your own framework. Document it well. Version control the control table schema. Make the parameter conventions explicit in a README. The framework you build is a durable asset — it's transferable across platform versions and new Microsoft product names, because the underlying pattern (Lookup → ForEach → parameterized child) will exist in whatever orchestration platform Microsoft ships next.
The ecosystem will produce Fabric-native metadata framework tooling within the next 12-18 months. Some community developer will build the Fabric equivalent of what BIML did for SSIS. When that happens, I'll write about it.
Until then: build it yourself. I'm happy to help you design it right. As always, reach out.