Azure Pipelines: Loops

Adam Burns

This is about Azure YAML Pipelines, not Azure Classic Pipelines. The docs explain the differences.

Everything shown here is in the docs, but the pieces are scattered and the syntax is fiddly. It took some digging and testing to figure out the options. This article collects all the findings.

In Azure Pipelines, the matrix job strategy and the each keyword both do loop operations. This article shows how to use them in five patterns that dynamically create jobs and steps, but you’re not limited to these examples. The each keyword, especially, can be adapted to many other cases.

Table of Contents

Jobs Created by a Hardcoded Matrix

Pipeline jobs with a matrix strategy dynamically create copies of themselves that each have different variables. This is essentially the same as looping over the matrix and creating one job for each set of those variables. Microsoft uses it for things like testing versions in parallel.

jobs:
- job: MatrixHardcoded
  pool:
    vmImage: ubuntu-20.04
  strategy:
    matrix:
      Thing1:
        thing: foo
      Thing2:
        thing: bar
  steps:
  - pwsh: Write-Output $(thing)
    displayName: Show thing

This creates MatrixHardcoded Thing1 and Matrix Hardcoded Thing2 jobs that each print the value of their thing variable in a Show thing step.

Jobs Created by an Each Loop over an Array

Pipelines have an each keyword in their expression syntax that implements loops more similar to what’s in programming languages like PowerShell and Python. Microsoft has great examples of its uses in their azure-pipelines-yaml repo.

parameters:
- name: thingsArray
  type: object
  default:
  - foo
  - bar

jobs:
- ${{each thing in parameters.thingsArray}}:
  - job: EachArrayJobsThing_${{thing}}
    pool:
      vmImage: ubuntu-20.04
    steps:
    - pwsh: Write-Output ${{thing}}
      displayName: Show thing

Fiddly details:

  • The ${{ }} syntax resolves into values. Since those values are prefixed with a dash (-), YAML interprets them as elements of an array. You need that dash on both the expression and job definition lines (highlighted). This feels like it’ll create an array of arrays that each contain one job, instead of a flat array of jobs, which seems like it would break. Maybe the pipeline interprets this syntax as a flat array, maybe it handles a nested one. Either way, you need both those dashes.
  • The each line has to end with a colon (:), but references to the ${{thing}} loop variable after it don’t.
  • Parameters are different from variables. Parameters support complex types (like arrays we can loop over). Variables are always strings.
  • If you need variables in your loop code, you can reference them in the expression syntax.
  • Parameters are mostly documented in the context of templates, but they can be used directly in pipelines.

This is mostly the same as a hardcoded matrix, but it creates jobs from a parameter that can be passed in dynamically.

There are some cosmetic differences. Since we used an array of values instead of a map of keys and values, there are no ThingN keys to use in the job names. They’re differentiated with values instead (foo and bar). The delimiters are underscores because job names don’t allow spaces (we could work around this with the displayName property).

We still get two jobs that each output their thing variable in a Show thing step.

Jobs Created by an Each Loop over a Map

This is the same as the previous pattern except it processes a map instead of an array.

parameters:
- name: thingsMap
  type: object
  default:
    Thing1: foo
    Thing2: bar

jobs:
- ${{each thing in parameters.thingsMap}}:
  - job: EachMapJobs${{thing.key}}
    pool:
      vmImage: ubuntu-20.04
    steps:
    - pwsh: Write-Output ${{thing.value}}
      displayName: Show thing

Since it’s processing a map, it references thing.key and thing.value instead of just thing. Again it creates two jobs with one step each.

Jobs Created by a Matrix Defined by an Each Loop over a Map

This combines the previous patterns to dynamically define a matrix using an each loop over a map parameter.

parameters:
- name: thingsMap
  type: object
  default:
    Thing1: foo
    Thing2: bar

jobs:
- job: MatrixEachMap
  pool:
    vmImage: ubuntu-20.04
  strategy:
    matrix:
      ${{each thing in parameters.thingsMap}}:
        ${{thing.key}}:
          thing: ${{thing.value}}
  steps:
  - pwsh: Write-Output $(thing)
    displayName: Show thing

Fiddly details:

  • We don’t need the YAML dashes (-) like we did in the two previous examples because we’re creating a map of config for the matrix, not an array of jobs. The ${{ }} syntax resolves to values that we want YAML to interpret as map keys, not array elements.
  • The each line still has to end with a colon (:).
  • We need a new colon (:) after ${{thing.key}} to tell YAML these are keys of a map.

The is the same as a hardcoded matrix except that its variables are dynamically referenced from a map parameter.

Steps with an Each Loop over an Array

The previous patterns used loops to dynamically create multiple jobs. This statically defines one job and dynamically creates multiple steps inside of it.

parameters:
- name: thingsArray
  type: object
  default:
  - foo
  - bar

jobs:
- job: EachArraySteps
  pool:
    vmImage: ubuntu-20.04
  steps:
  - ${{each thing in parameters.thingsArray}}:
    - pwsh: Write-Output ${{thing}}
      displayName: Show thing

As expected, we get one job that contains two Show thing steps.

The differences between these patterns are syntactically small, but they give you a lot of implementation options. Hopefully these examples help you find one that work for your use case.

Happy automating!

Operating Ops

Need more than just this article? We’re available to consult.

You might also want to check out these related articles: