ADF ARM Template CI/CD: The Complete Pattern After Three Years of Refinement

Shannon Lowder

Shannon Lowder

— 2 min read

I have been iterating on the ADF CI/CD pattern since 2018 when git integration shipped. I've rebuilt it for five different clients, hit the same problems each time, and landed on a pattern that I now consider stable. Here it is -- the complete picture, including the parts Microsoft's documentation glosses over.

The Architecture

Three environments: Dev, Test, Prod. Each is a separate ADF instance with its own linked services pointing to environment-appropriate resources (different connection strings, different storage accounts, different SQL servers).

Git integration is configured on Dev only. Developers work in Dev ADF Studio, push feature branches to Azure Repos (or GitHub), and open PRs against the collaboration branch (I use main). PR reviews happen in Git. Merges go to main.

When a developer is ready to deploy to Test/Prod, they click Publish in Dev ADF Studio. This generates ARM templates and writes them to the adf_publish branch. An Azure DevOps pipeline triggers on changes to adf_publish and runs the multi-stage deployment.

Test and Prod ADF instances are not connected to git -- they receive ARM template deployments only. This is intentional. You don't want someone hand-editing pipelines in Prod ADF Studio and having those changes overwritten on the next deployment.

The ADO Pipeline Structure

trigger:
  branches:
    include:
      - adf_publish

stages:
  - stage: DeployTest
    jobs:
      - job: Deploy
        steps:
          - task: AzurePowerShell@5
            displayName: Pre-deployment (stop triggers)
            inputs:
              azureSubscription: service-connection-test
              ScriptPath: $(Build.Repository.LocalPath)/adf-cicd/pre-deploy.ps1

          - task: AzureResourceManagerTemplateDeployment@3
            displayName: Deploy ARM Template
            inputs:
              deploymentScope: Resource Group
              azureResourceManagerConnection: service-connection-test
              resourceGroupName: $(testResourceGroup)
              csmFile: adf_publish/$(devAdfName)/ARMTemplateForFactory.json
              csmParametersFile: adf-cicd/parameters/test.parameters.json

          - task: AzurePowerShell@5
            displayName: Post-deployment (restart triggers)
            inputs:
              azureSubscription: service-connection-test
              ScriptPath: $(Build.Repository.LocalPath)/adf-cicd/post-deploy.ps1

  - stage: DeployProd
    dependsOn: DeployTest
    condition: succeeded()

The Pre-Deployment Script

Triggers must be stopped before ARM deployment. If you deploy while triggers are running, the deployment fails or leaves triggers in an inconsistent state. This is not documented prominently enough.

param(
    [string]$ResourceGroupName,
    [string]$DataFactoryName
)

$triggers = Get-AzDataFactoryV2Trigger `
    -ResourceGroupName $ResourceGroupName `
    -DataFactoryName $DataFactoryName

foreach ($trigger in $triggers) {
    if ($trigger.RuntimeState -eq "Started") {
        Write-Host "Stopping trigger: $($trigger.Name)"
        Stop-AzDataFactoryV2Trigger `
            -ResourceGroupName $ResourceGroupName `
            -DataFactoryName $DataFactoryName `
            -Name $trigger.Name `
            -Force
    }
}
Write-Host "All triggers stopped."

The Post-Deployment Script

After ARM deployment, restart only the triggers that should be active in this environment.

param(
    [string]$ResourceGroupName,
    [string]$DataFactoryName,
    [string[]]$TriggerNames
)

foreach ($triggerName in $TriggerNames) {
    Write-Host "Starting trigger: $triggerName"
    Start-AzDataFactoryV2Trigger `
        -ResourceGroupName $ResourceGroupName `
        -DataFactoryName $DataFactoryName `
        -Name $triggerName `
        -Force
}
Write-Host "Triggers started."

The Environment Parameter File Pattern

The ARM template generated by ADF Publish includes a parameters section for every linked service connection string, every global parameter, and every integration runtime reference. The environment parameter file overrides these with environment-specific values.

Keep this file in the repo under adf-cicd/parameters/, one file per environment. Never hardcode connection strings in the ADF UI -- always use global parameters that get overridden by the environment parameter file.

{
  "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "AzureSqlDatabase_connectionString": {
      "value": "Server=tcp:prod-sql.database.windows.net;Database=prod-db;..."
    }
  }
}

The One Remaining Rough Edge

The Publish button is still manual. When your team merges a PR to main, the collaboration branch is up to date, but adf_publish is not. Someone has to go to Dev ADF Studio, verify they're on main, and click Publish. There is no way to automate this step with built-in ADF functionality.

The community workaround is the microsoft/azure-data-factory-utilities npm package, which exposes a programmatic publish API. I've used it in one engagement and it works, but it's community tooling, not a Microsoft-supported feature. I'll write a dedicated post on automated publish when I have more production miles on it.

Questions about wiring this up for your ADF deployment? I'm here to help.

Read more