Testing Azure Pipeline Artifacts

Azure Pipelines supports several types of artifacts. This is about Pipeline Artifacts, the ones managed with the publish and download tasks.

Any pipeline that builds code should do at least these two things:

  • Build an artifact that can be deployed later.
  • Test that artifact.

Specifically, it should test the artifact. Not just the same version of application code used to build the artifact, but the actual artifact file that was built. If it only tests the same version of code, it won’t detect bugs in how that code was built.

This is a case where more errors are better. If we hide errors at build time, we’ll have to diagnose them at deploy time. That could cause release failures, and maybe outages. Like the Zen of Python says:

Errors should never pass silently.

Testing the artifact itself is a best practice for any build pipeline. It’s better to find out right away that the code wasn’t built correctly.

First we’ll create a pipeline that tests its code but not the artifact it builds. We’ll include an artificial bug in the build process and show that the pipeline passes tests but still creates a broken artifact. Then we’ll rework to point the tests at the artifact so the build bug gets caught by the tests and becomes visible.

These examples use Python’s tox, but the principles are the same for any tooling. Tox is a testing tool that creates isolated Python environments, installs the app being tested into those environments, and runs test commands.

Setup

First we need a Python package. We’ll make a super-simple one called app:

.
├── app
│   ├── __init__.py
│   └── main.py
├── setup.py
└── tox.ini

app is a single package with an empty __init__.py. The package contains one main.py module that defines one function:

def main():
    print('Success!')

setup.py contains config that lets us build app into a Python wheel file (an artifact that can be installed into a Python environment):

from setuptools import setup

setup(
    author='Operating Ops, LLC',
    license='MIT',
    description='Demo app.',
    name='app',
    packages=['app'],
    version='0.0.1'
)

tox.ini tells tox to call our main() function:

[tox]
envlist = py38

[testenv]
commands = python -c 'from app.main import main; main()'

That’s not a real test, but it’ll be enough to show the difference between exercising source code and built artifacts. A real project would use the unittest library or pytest or another tool here.

This test passes locally:

(env3) PS /Users/adam/Local/laboratory/pipelines/testing_pipeline_artifacts> tox -e py38
GLOB sdist-make: /Users/adam/Local/laboratory/pipelines/testing_pipeline_artifacts/setup.py
py38 recreate: /Users/adam/Local/laboratory/pipelines/testing_pipeline_artifacts/.tox/py38
py38 inst: /Users/adam/Local/laboratory/pipelines/testing_pipeline_artifacts/.tox/.tmp/package/1/app-0.0.1.zip
py38 installed: app @ file:///Users/adam/Local/laboratory/pipelines/testing_pipeline_artifacts/.tox/.tmp/package/1/app-0.0.1.zip
py38 run-test-pre: PYTHONHASHSEED='3356214888'
py38 run-test: commands[0] | python -c 'from app.main import main; main()'
Success!
___________________________________________________________________________ summary ____________________________________________________________________________
  py38: commands succeeded
  congratulations :)

Negative Case

Our code works locally, now we need a build pipeline to make an artifact we can deploy. We’ll start with the negative case, a broken build that still passes tests:

jobs:
- job: Build
  pool:
    vmImage: ubuntu-20.04
  workspace:
    clean: outputs
  steps:
  - task: UsePythonVersion@0
    displayName: Use Python 3.8
    inputs:
      versionSpec: '3.8'
  - pwsh: pip install --upgrade pip setuptools wheel
    displayName: Install build tools
  - pwsh: Remove-Item app/main.py
    workingDirectory: $(Build.SourcesDirectory)/pipelines/testing_pipeline_artifacts
    displayName: BUILD BUG
  - pwsh: python setup.py bdist_wheel --dist-dir $(Build.BinariesDirectory)
    workingDirectory: $(Build.SourcesDirectory)/pipelines/testing_pipeline_artifacts
    displayName: Build wheel
  - publish: $(Build.BinariesDirectory)/app-0.0.1-py3-none-any.whl
    displayName: Publish wheel
    artifact: wheel

- job: Test
  dependsOn: Build
  pool:
    vmImage: ubuntu-20.04
  steps:
  - task: UsePythonVersion@0
    displayName: Use Python 3.8
    inputs:
      versionSpec: '3.8'
  - pwsh: pip install --upgrade tox
    displayName: Install tox

    # This tests the version of code used to build the 'wheel' artifact, but it
    # doesn't test the artifact itself.
  - pwsh: tox -e py38
    workingDirectory: $(Build.SourcesDirectory)/pipelines/testing_pipeline_artifacts
    displayName: Run tox on code

The Build and Test jobs both succeed even though the BUILD BUG task ran:

An artifact is published to the pipeline:

We can download it:

But, if we install it and try to import from app, we get errors:

(env3) PS /Users/adam/Downloads> pip install ./app-0.0.1-py3-none-any.whl
Processing ./app-0.0.1-py3-none-any.whl
Installing collected packages: app
Successfully installed app-0.0.1
(env3) PS /Users/adam/Downloads> python
Python 3.8.3 (default, Jul  1 2020, 07:50:15) 
[Clang 11.0.0 (clang-1100.0.33.16)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from app.main import main
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'app.main'

It couldn’t find app.main because that module doesn’t exist. Our BUILD BUG task deleted it before the artifact was built. Later, the Test job checked out a fresh copy of the code, which included a fresh copy of the file we accidentally deleted in the Build job. Tox ran in that fresh environment and passed because all the files it needed were present. It was testing the code from the repo, not the artifact we built.

The Fix

If the artifact created by the pipeline doesn’t work, the pipeline should fail. To make tox test the app-0.0.1-py3-none-any.whl file built in the Build job, we need to do two things:

  • Download the artifact in the Test job.
  • Tell tox to test that artifact instead of the files from the repo. Normally, tox builds its own artifacts from source when it runs (that’s what you want when you’re testing locally). We can override this and tell it to install our pipeline’s artifact with the --installpkg flag.

First we need to modify the Test job from our pipeline:

- job: Test
  dependsOn: Build
  pool:
    vmImage: ubuntu-20.04
  steps:
  - task: UsePythonVersion@0
    displayName: Use Python 3.8
    inputs:
      versionSpec: '3.8'
  - pwsh: pip install --upgrade tox
    displayName: Install tox
  - download: current
    displayName: Download wheel
    artifact: wheel

    # This tests the version of code used to build the 'wheel' artifact, but it
    # doesn't test the artifact itself.
  - pwsh: tox -e py38
    workingDirectory: $(Build.SourcesDirectory)/pipelines/testing_pipeline_artifacts
    displayName: Run tox on code

    # This tests the artifact built in the build job above.
    # https://tox.readthedocs.io/en/latest/config.html#conf-sdistsrc
  - pwsh: tox --installpkg $(Pipeline.Workspace)/wheel/app-0.0.1-py3-none-any.whl -e py38
    workingDirectory: $(Build.SourcesDirectory)/pipelines/testing_pipeline_artifacts
    displayName: Run tox on artifact

We kept the old (invalid) test so we can compare it to the new test in the next run.

These two changes are needed for pretty much any build and test system. For Python and tox in this lab specifically, we also need to:

  • Recreate the Python environments between tests. In the “Run tox on code” task, tox will automatically build and install version 0.0.1 of app from the code in the repo. Unless we get rid of that, the “Run tox on artifact” task will see that version 0.0.1 of the app is already installed, so it won’t install the artifact we pass with --installpkg.
  • Change directory away from the repo root. Otherwise the test may import files from the current directory instead of the artifact we pass with --installpkg.

We can do this with two changes to tox.ini:

[tox]
envlist = py38

[testenv]
# Recreate venvs so previously-installed packages aren't importable.
# https://tox.readthedocs.io/en/latest/config.html#conf-recreate
recreate = true

# Change directory so packages in the current directory aren't importable.
# https://tox.readthedocs.io/en/latest/config.html#conf-changedir
# It's convenient to use the {toxworkdir}, but other directories work.
# https://tox.readthedocs.io/en/latest/config.html#globally-available-substitutions
changedir = {toxworkdir}

commands = python -c 'from app.main import main; main()'

The new test fails:

We get the same ModuleNotFoundError we got when we tried to install the artifact manually and import from it. That shows us the new test is exercising the artifact built in the Build job, not just the code that’s in the repo.

Now when there’s a bug in the build, the pipeline will fail at build time. Fixes can be engineered before release, and broken artifacts won’t go live.

Happy building!

Operating Ops

Need more than just this article? We’re available to consult.

You might also want to check out these related articles: