ADF CI/CD with ARM Templates: Shipping Pipeline Changes Like Real Code

Shannon Lowder

Shannon Lowder

— 3 min read

Git integration landed. Now let's talk about what you do with it. Because "pipelines in git" is table stakes — what you actually want is a deployment pipeline that takes your ADF changes from development to production the same way you ship application code.

Here's the complete CI/CD workflow I've settled on for ADF v2 with Azure DevOps.

The Deployment Artifact: adf_publish

As I covered in the git integration post, when you click Publish in the ADF UI, it generates ARM templates to the adf_publish branch. This branch is your deployment artifact. Your ADO pipeline deploys from here.

The adf_publish branch contains two files:

  • ARMTemplateForFactory.json — the full ARM template with all resources
  • ARMTemplateParametersForFactory.json — the default parameters file

The parameters file contains the linked service connection strings, storage account URLs, and other environment-specific values. You'll create separate parameter files per environment to override these values at deploy time.

The Full Workflow

Step 1: Feature Branch Development

Developer branches from main in the ADF UI or via git. Makes changes to pipelines, datasets, linked services. Saves in ADF UI (commits to feature branch). Submits PR to main. Team reviews the pipeline JSON diffs in the PR. PR is merged.

Step 2: Publish

Someone with appropriate permissions clicks Publish in the ADF UI (pointed at the dev ADF instance, linked to main branch). This generates the ARM templates to adf_publish. This is the trigger for your deployment pipeline.

Step 3: ADO Pipeline Deploys to Dev

Your ADO pipeline detects changes to the adf_publish branch and begins the deployment stages. First stage: deploy to the dev ADF instance (which is the same instance you're authoring against, so this is technically a no-op validation step for dev). In practice, some teams skip the dev deploy stage and start at test.

Step 4: Deploy to Test and Prod

Subsequent stages promote to test and prod ADF instances using the same ARM template but with environment-specific parameter files.

The Pre/Post Deployment Scripts

Before you ARM-deploy to a target ADF instance, you need to stop all triggers. ARM deployment recreates resources — a trigger that's running during deployment can cause duplicate pipeline runs or deployment failures.

Here's the PowerShell for trigger management:

# Pre-deployment: Stop all triggers
param(
    [string]$ResourceGroupName,
    [string]$DataFactoryName,
    [string]$ArmTemplate
)

$template = Get-Content $ArmTemplate | ConvertFrom-Json

# Get triggers defined in the ARM template
$triggersInTemplate = $template.resources |
    Where-Object { $_.type -eq "Microsoft.DataFactory/factories/triggers" }

# Stop each trigger that's currently running
foreach ($trigger in $triggersInTemplate) {
    $triggerName = $trigger.name.Split('/')[1]
    $triggerResource = Get-AzDataFactoryV2Trigger `
        -ResourceGroupName $ResourceGroupName `
        -DataFactoryName $DataFactoryName `
        -Name $triggerName `
        -ErrorAction SilentlyContinue

    if ($triggerResource -and $triggerResource.RuntimeState -eq 'Started') {
        Write-Host "Stopping trigger: $triggerName"
        Stop-AzDataFactoryV2Trigger `
            -ResourceGroupName $ResourceGroupName `
            -DataFactoryName $DataFactoryName `
            -Name $triggerName `
            -Force
    }
}

Write-Host "Pre-deployment trigger stop complete"

After deployment, restart the triggers:

# Post-deployment: Start triggers
foreach ($trigger in $triggersInTemplate) {
    $triggerName = $trigger.name.Split('/')[1]
    Write-Host "Starting trigger: $triggerName"
    Start-AzDataFactoryV2Trigger `
        -ResourceGroupName $ResourceGroupName `
        -DataFactoryName $DataFactoryName `
        -Name $triggerName `
        -Force
}

Write-Host "Post-deployment trigger start complete"

The ADO Pipeline YAML

trigger:
  branches:
    include:
      - adf_publish

variables:
  resourceGroup: 'rg-data-platform'
  devFactory: 'adf-dataplatform-dev'
  testFactory: 'adf-dataplatform-test'
  prodFactory: 'adf-dataplatform-prod'

stages:
  - stage: DeployTest
    displayName: 'Deploy to Test'
    jobs:
      - deployment: DeployADF
        environment: 'test'
        strategy:
          runOnce:
            deploy:
              steps:
                - checkout: self

                - task: AzurePowerShell@5
                  displayName: 'Stop Triggers (Test)'
                  inputs:
                    azureSubscription: 'AzureServiceConnection'
                    ScriptType: 'FilePath'
                    ScriptPath: '$(Build.SourcesDirectory)/scripts/PreDeployment.ps1'
                    ScriptArguments: >
                      -ResourceGroupName $(resourceGroup)
                      -DataFactoryName $(testFactory)
                      -ArmTemplate '$(Build.SourcesDirectory)/ARMTemplateForFactory.json'
                    azurePowerShellVersion: 'LatestVersion'

                - task: AzureResourceManagerTemplateDeployment@3
                  displayName: 'Deploy ARM Template (Test)'
                  inputs:
                    deploymentScope: 'Resource Group'
                    azureResourceManagerConnection: 'AzureServiceConnection'
                    subscriptionId: '$(subscriptionId)'
                    resourceGroupName: '$(resourceGroup)'
                    location: 'East US 2'
                    templateLocation: 'Linked artifact'
                    csmFile: '$(Build.SourcesDirectory)/ARMTemplateForFactory.json'
                    csmParametersFile: '$(Build.SourcesDirectory)/ARMTemplateParametersForFactory.test.json'
                    deploymentMode: 'Incremental'

                - task: AzurePowerShell@5
                  displayName: 'Start Triggers (Test)'
                  inputs:
                    azureSubscription: 'AzureServiceConnection'
                    ScriptType: 'FilePath'
                    ScriptPath: '$(Build.SourcesDirectory)/scripts/PostDeployment.ps1'
                    ScriptArguments: >
                      -ResourceGroupName $(resourceGroup)
                      -DataFactoryName $(testFactory)
                      -ArmTemplate '$(Build.SourcesDirectory)/ARMTemplateForFactory.json'
                    azurePowerShellVersion: 'LatestVersion'

  - stage: DeployProd
    displayName: 'Deploy to Production'
    dependsOn: DeployTest
    condition: succeeded()
    jobs:
      - deployment: DeployADF
        environment: 'production'
        strategy:
          runOnce:
            deploy:
              steps:
                # Same pattern as Test, with prod factory name and prod parameter file

The Parameter File Pattern

Create one parameter file per environment:

  • ARMTemplateParametersForFactory.test.json
  • ARMTemplateParametersForFactory.prod.json

Each file overrides the environment-specific values: linked service connection strings, storage account names, SQL server hostnames. The ARM template deploys the same pipeline logic everywhere; the parameter files inject the environment-specific connections.

Store the actual credentials (connection strings, keys) in Azure Key Vault, not in the parameter files. The parameter files reference Key Vault secret URIs. Your ADF linked services use AKV references, not inline credentials.

The Gotcha: ARM Template Size

For a factory with 40+ pipelines, the generated ARM template can exceed 4MB. Azure Resource Manager has a 4MB template size limit. When you hit this, the deployment fails with an unhelpful error.

The workaround: use ARM template linking, where you split the factory into multiple templates linked from a parent. ADF doesn't generate linked templates automatically — you'll need a post-processing script that splits the generated ARM template. Not elegant, but manageable.

Trust me on this one: you'll hit this limit eventually if your factory grows. Plan for it before it hits you in a late-night deployment.

This is real infrastructure-as-code for ADF pipelines. The path exists now. Walk it. As always, I'm here to help if you get stuck on the trigger management scripts or the parameter file pattern.

Read more