Testing Azure Pipeline Artifacts

Azure Pipelines supports several types of artifacts. This is about Pipeline Artifacts, the ones managed with the publish and download tasks.

Any pipeline that builds code should do at least these two things:

  • Build an artifact that can be deployed later.
  • Test that artifact.

Specifically, it should test the artifact. Not just the same version of application code used to build the artifact, but the actual artifact file that was built. If it only tests the same version of code, it won’t detect bugs in how that code was built.

This is a case where more errors are better. If we hide errors at build time, we’ll have to diagnose them at deploy time. That could cause release failures, and maybe outages. Like the Zen of Python says:

Errors should never pass silently.

Testing the artifact itself is a best practice for any build pipeline. It’s better to find out right away that the code wasn’t built correctly.

First we’ll create a pipeline that tests its code but not the artifact it builds. We’ll include an artificial bug in the build process and show that the pipeline passes tests but still creates a broken artifact. Then we’ll rework to point the tests at the artifact so the build bug gets caught by the tests and becomes visible.

These examples use Python’s tox, but the principles are the same for any tooling. Tox is a testing tool that creates isolated Python environments, installs the app being tested into those environments, and runs test commands.

Setup

First we need a Python package. We’ll make a super-simple one called app:

.
├── app
│   ├── __init__.py
│   └── main.py
├── setup.py
└── tox.ini

app is a single package with an empty __init__.py. The package contains one main.py module that defines one function:

def main():
    print('Success!')

setup.py contains config that lets us build app into a Python wheel file (an artifact that can be installed into a Python environment):

from setuptools import setup

setup(
    author='Operating Ops, LLC',
    license='MIT',
    description='Demo app.',
    name='app',
    packages=['app'],
    version='0.0.1'
)

tox.ini tells tox to call our main() function:

[tox]
envlist = py38

[testenv]
commands = python -c 'from app.main import main; main()'

That’s not a real test, but it’ll be enough to show the difference between exercising source code and built artifacts. A real project would use the unittest library or pytest or another tool here.

This test passes locally:

(env3) PS /Users/adam/Local/laboratory/pipelines/testing_pipeline_artifacts> tox -e py38
GLOB sdist-make: /Users/adam/Local/laboratory/pipelines/testing_pipeline_artifacts/setup.py
py38 recreate: /Users/adam/Local/laboratory/pipelines/testing_pipeline_artifacts/.tox/py38
py38 inst: /Users/adam/Local/laboratory/pipelines/testing_pipeline_artifacts/.tox/.tmp/package/1/app-0.0.1.zip
py38 installed: app @ file:///Users/adam/Local/laboratory/pipelines/testing_pipeline_artifacts/.tox/.tmp/package/1/app-0.0.1.zip
py38 run-test-pre: PYTHONHASHSEED='3356214888'
py38 run-test: commands[0] | python -c 'from app.main import main; main()'
Success!
___________________________________________________________________________ summary ____________________________________________________________________________
  py38: commands succeeded
  congratulations :)

Negative Case

Our code works locally, now we need a build pipeline to make an artifact we can deploy. We’ll start with the negative case, a broken build that still passes tests:

jobs:
- job: Build
  pool:
    vmImage: ubuntu-20.04
  workspace:
    clean: outputs
  steps:
  - task: UsePythonVersion@0
    displayName: Use Python 3.8
    inputs:
      versionSpec: '3.8'
  - pwsh: pip install --upgrade pip setuptools wheel
    displayName: Install build tools
  - pwsh: Remove-Item app/main.py
    workingDirectory: $(Build.SourcesDirectory)/pipelines/testing_pipeline_artifacts
    displayName: BUILD BUG
  - pwsh: python setup.py bdist_wheel --dist-dir $(Build.BinariesDirectory)
    workingDirectory: $(Build.SourcesDirectory)/pipelines/testing_pipeline_artifacts
    displayName: Build wheel
  - publish: $(Build.BinariesDirectory)/app-0.0.1-py3-none-any.whl
    displayName: Publish wheel
    artifact: wheel

- job: Test
  dependsOn: Build
  pool:
    vmImage: ubuntu-20.04
  steps:
  - task: UsePythonVersion@0
    displayName: Use Python 3.8
    inputs:
      versionSpec: '3.8'
  - pwsh: pip install --upgrade tox
    displayName: Install tox

    # This tests the version of code used to build the 'wheel' artifact, but it
    # doesn't test the artifact itself.
  - pwsh: tox -e py38
    workingDirectory: $(Build.SourcesDirectory)/pipelines/testing_pipeline_artifacts
    displayName: Run tox on code

The Build and Test jobs both succeed even though the BUILD BUG task ran:

An artifact is published to the pipeline:

We can download it:

But, if we install it and try to import from app, we get errors:

(env3) PS /Users/adam/Downloads> pip install ./app-0.0.1-py3-none-any.whl
Processing ./app-0.0.1-py3-none-any.whl
Installing collected packages: app
Successfully installed app-0.0.1
(env3) PS /Users/adam/Downloads> python
Python 3.8.3 (default, Jul  1 2020, 07:50:15) 
[Clang 11.0.0 (clang-1100.0.33.16)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from app.main import main
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'app.main'

It couldn’t find app.main because that module doesn’t exist. Our BUILD BUG task deleted it before the artifact was built. Later, the Test job checked out a fresh copy of the code, which included a fresh copy of the file we accidentally deleted in the Build job. Tox ran in that fresh environment and passed because all the files it needed were present. It was testing the code from the repo, not the artifact we built.

The Fix

If the artifact created by the pipeline doesn’t work, the pipeline should fail. To make tox test the app-0.0.1-py3-none-any.whl file built in the Build job, we need to do two things:

  • Download the artifact in the Test job.
  • Tell tox to test that artifact instead of the files from the repo. Normally, tox builds its own artifacts from source when it runs (that’s what you want when you’re testing locally). We can override this and tell it to install our pipeline’s artifact with the --installpkg flag.

First we need to modify the Test job from our pipeline:

- job: Test
  dependsOn: Build
  pool:
    vmImage: ubuntu-20.04
  steps:
  - task: UsePythonVersion@0
    displayName: Use Python 3.8
    inputs:
      versionSpec: '3.8'
  - pwsh: pip install --upgrade tox
    displayName: Install tox
  - download: current
    displayName: Download wheel
    artifact: wheel

    # This tests the version of code used to build the 'wheel' artifact, but it
    # doesn't test the artifact itself.
  - pwsh: tox -e py38
    workingDirectory: $(Build.SourcesDirectory)/pipelines/testing_pipeline_artifacts
    displayName: Run tox on code

    # This tests the artifact built in the build job above.
    # https://tox.readthedocs.io/en/latest/config.html#conf-sdistsrc
  - pwsh: tox --installpkg $(Pipeline.Workspace)/wheel/app-0.0.1-py3-none-any.whl -e py38
    workingDirectory: $(Build.SourcesDirectory)/pipelines/testing_pipeline_artifacts
    displayName: Run tox on artifact

We kept the old (invalid) test so we can compare it to the new test in the next run.

These two changes are needed for pretty much any build and test system. For Python and tox in this lab specifically, we also need to:

  • Recreate the Python environments between tests. In the “Run tox on code” task, tox will automatically build and install version 0.0.1 of app from the code in the repo. Unless we get rid of that, the “Run tox on artifact” task will see that version 0.0.1 of the app is already installed, so it won’t install the artifact we pass with --installpkg.
  • Change directory away from the repo root. Otherwise the test may import files from the current directory instead of the artifact we pass with --installpkg.

We can do this with two changes to tox.ini:

[tox]
envlist = py38

[testenv]
# Recreate venvs so previously-installed packages aren't importable.
# https://tox.readthedocs.io/en/latest/config.html#conf-recreate
recreate = true

# Change directory so packages in the current directory aren't importable.
# https://tox.readthedocs.io/en/latest/config.html#conf-changedir
# It's convenient to use the {toxworkdir}, but other directories work.
# https://tox.readthedocs.io/en/latest/config.html#globally-available-substitutions
changedir = {toxworkdir}

commands = python -c 'from app.main import main; main()'

The new test fails:

We get the same ModuleNotFoundError we got when we tried to install the artifact manually and import from it. That shows us the new test is exercising the artifact built in the Build job, not just the code that’s in the repo.

Now when there’s a bug in the build, the pipeline will fail at build time. Fixes can be engineered before release, and broken artifacts won’t go live.

Happy building!

Operating Ops

Need more than just this article? We’re available to consult.

You might also want to check out these related articles:

Azure Pipelines: Loops

This is about Azure YAML Pipelines, not Azure Classic Pipelines. The docs explain the differences.

Everything shown here is in the docs, but the pieces are scattered and the syntax is fiddly. It took some digging and testing to figure out the options. This article collects all the findings.

In Azure Pipelines, the matrix job strategy and the each keyword both do loop operations. This article shows how to use them in five patterns that dynamically create jobs and steps, but you’re not limited to these examples. The each keyword, especially, can be adapted to many other cases.

Table of Contents

Jobs Created by a Hardcoded Matrix

Pipeline jobs with a matrix strategy dynamically create copies of themselves that each have different variables. This is essentially the same as looping over the matrix and creating one job for each set of those variables. Microsoft uses it for things like testing versions in parallel.

jobs:
- job: MatrixHardcoded
  pool:
    vmImage: ubuntu-20.04
  strategy:
    matrix:
      Thing1:
        thing: foo
      Thing2:
        thing: bar
  steps:
  - pwsh: Write-Output $(thing)
    displayName: Show thing

This creates MatrixHardcoded Thing1 and Matrix Hardcoded Thing2 jobs that each print the value of their thing variable in a Show thing step.

Jobs Created by an Each Loop over an Array

Pipelines have an each keyword in their expression syntax that implements loops more similar to what’s in programming languages like PowerShell and Python. Microsoft has great examples of its uses in their azure-pipelines-yaml repo.

parameters:
- name: thingsArray
  type: object
  default:
  - foo
  - bar

jobs:
- ${{each thing in parameters.thingsArray}}:
  - job: EachArrayJobsThing_${{thing}}
    pool:
      vmImage: ubuntu-20.04
    steps:
    - pwsh: Write-Output ${{thing}}
      displayName: Show thing

Fiddly details:

  • The ${{ }} syntax resolves into values. Since those values are prefixed with a dash (-), YAML interprets them as elements of an array. You need that dash on both the expression and job definition lines (highlighted). This feels like it’ll create an array of arrays that each contain one job, instead of a flat array of jobs, which seems like it would break. Maybe the pipeline interprets this syntax as a flat array, maybe it handles a nested one. Either way, you need both those dashes.
  • The each line has to end with a colon (:), but references to the ${{thing}} loop variable after it don’t.
  • Parameters are different from variables. Parameters support complex types (like arrays we can loop over). Variables are always strings.
  • If you need variables in your loop code, you can reference them in the expression syntax.
  • Parameters are mostly documented in the context of templates, but they can be used directly in pipelines.

This is mostly the same as a hardcoded matrix, but it creates jobs from a parameter that can be passed in dynamically.

There are some cosmetic differences. Since we used an array of values instead of a map of keys and values, there are no ThingN keys to use in the job names. They’re differentiated with values instead (foo and bar). The delimiters are underscores because job names don’t allow spaces (we could work around this with the displayName property).

We still get two jobs that each output their thing variable in a Show thing step.

Jobs Created by an Each Loop over a Map

This is the same as the previous pattern except it processes a map instead of an array.

parameters:
- name: thingsMap
  type: object
  default:
    Thing1: foo
    Thing2: bar

jobs:
- ${{each thing in parameters.thingsMap}}:
  - job: EachMapJobs${{thing.key}}
    pool:
      vmImage: ubuntu-20.04
    steps:
    - pwsh: Write-Output ${{thing.value}}
      displayName: Show thing

Since it’s processing a map, it references thing.key and thing.value instead of just thing. Again it creates two jobs with one step each.

Jobs Created by a Matrix Defined by an Each Loop over a Map

This combines the previous patterns to dynamically define a matrix using an each loop over a map parameter.

parameters:
- name: thingsMap
  type: object
  default:
    Thing1: foo
    Thing2: bar

jobs:
- job: MatrixEachMap
  pool:
    vmImage: ubuntu-20.04
  strategy:
    matrix:
      ${{each thing in parameters.thingsMap}}:
        ${{thing.key}}:
          thing: ${{thing.value}}
  steps:
  - pwsh: Write-Output $(thing)
    displayName: Show thing

Fiddly details:

  • We don’t need the YAML dashes (-) like we did in the two previous examples because we’re creating a map of config for the matrix, not an array of jobs. The ${{ }} syntax resolves to values that we want YAML to interpret as map keys, not array elements.
  • The each line still has to end with a colon (:).
  • We need a new colon (:) after ${{thing.key}} to tell YAML these are keys of a map.

The is the same as a hardcoded matrix except that its variables are dynamically referenced from a map parameter.

Steps with an Each Loop over an Array

The previous patterns used loops to dynamically create multiple jobs. This statically defines one job and dynamically creates multiple steps inside of it.

parameters:
- name: thingsArray
  type: object
  default:
  - foo
  - bar

jobs:
- job: EachArraySteps
  pool:
    vmImage: ubuntu-20.04
  steps:
  - ${{each thing in parameters.thingsArray}}:
    - pwsh: Write-Output ${{thing}}
      displayName: Show thing

As expected, we get one job that contains two Show thing steps.

The differences between these patterns are syntactically small, but they give you a lot of implementation options. Hopefully these examples help you find one that work for your use case.

Happy automating!

Operating Ops

Need more than just this article? We’re available to consult.

You might also want to check out these related articles:

Azure Pipelines: If Statements

Hello!

This is about Azure YAML Pipelines, not Azure Classic Pipelines. The docs explain the differences.

Everything I show here is in the docs but the pieces are scattered and it took some work to find everything. This article collects what I found.

Pipelines support two kinds of conditions. The first isn’t an if statement, but it acts like one by letting you use expressions to choose when jobs, steps, or stages run. Like this example from the doc:

jobs:
- job: Foo
  steps:
  - script: echo Hello!
    condition: always()

There are a bunch of different functions you can use.

These conditions are handy but haven’t been enough on their own. It’s not enough to turn jobs on and off, I need to automate decisions about what config to pass them. Fortunately, you can also conditionally set variables and parameters using if statements. The expressions doc calls it “conditional insertion” and gives examples like this:

variables:
  ${{ if eq(variables['Build.SourceBranchName'], 'master') }}:
    stageName: prod

And this:

steps:
- task: PublishPipelineArtifact@1
  inputs:
    targetPath: '$(Pipeline.Workspace)'
    ${{ if eq(variables['Build.SourceBranchName'], 'master') }}:
      artifact: 'prod'

You can also use these in pipeline templates. The templates doc has its own section on conditional insertion that gives examples of lists of tasks:

steps:
- ${{ if eq(parameters.toolset, 'msbuild') }}:
  - task: msbuild@1
  - task: vstest@2

Last, Microsoft has a repo of example pipeline YAML and its each expressions also happen to show if statements. I’m linking the latest commit (at the time I wrote this) because it’s technically not a doc about if statements and future versions might not use the same examples. You may also want to check out the latest version.

DevOps is all about managing config and conditional variables and parameters are essential to doing that well. Most of the pipelines I build rely on this second kind of condition, the if statement. Hopefully it helps with yours.

Happy automating!

Adam

Need more than just this article? We’re available to consult.

You might also want to check out these related articles: