Staged execution of tests in Azure DevOps Pipelines

When running Unit Tests in your build system, you may want to first run the most important suite, the one that should never fail, the ones that are currently being changed, finally the regression suite and integration tests that may be slower to execute.

The Visual Studio Test task supports this feature, though many probably never knew that this was in the system since the first release of the Workflow based Team Build 2010.

In your build definition you can easily add multiple test runs, and for each test run define which tests should run. Not just fixed list of tests you have to curate, but a dynamic list based on attributes of the test.

The following steps will take you through the basic gist to get this working in your Azure Pipeline.

1. Categorize your tests

Step one is to group tests together that need go run together. This can be done in many different ways. Grouping can be done at the assembly or namespace level, yet the most flexible system I tend to rely on is grouping by Test Category.

Go through your test base and apply categories to your automated tests. Use the [TestCategory] attribute for MsTest and the [Category] attribute for NUnit and [Trait] for xUnit.

// MsTest
[TestCategory("Critical")]
[TestMethod]
public void MyCriticalTest {}

// NUnit
[Category("Critical")]
[Test]
public void MyCriticalTest {}

// xUnit
[Trait("Category", "Critical")]
[Fact]
public void MyCriticalTest {}

The Test Filter options for Visual Studio Test can also filter on other characteristics of your test, such as the name of the test, it's parent class, namespace or assembly. You can combine these elements:

// MsTest
[TestCategory("Critical")]
[TestPriority(1)]
[TestMethod]
public void MyCriticalTest {}

2. Create multiple test runs in Azure Pipelines

Once you've categorized your tests create the test runs in Azure Pipelines. Each stage or set of tests will map to its own test run. In this example we're going to distinguish critical tests from non-critical tests using a simple attribute.

In your Azure Pipeline add multiple Visual Studio Test tasks, one for each set of tests you want to run:

Add a Visual Studio Test task for each set of tests to run.

We're going to configure the first run to execute the critical tests (and fail the build when any of these tests fail). Only when these tests succeed will we run the non-critical tests.

In the VsTest - Critical Tests configure the test filter to only run Critical tests:

Set the Test filter Criteria to TestCategory=Critical

Name the test run so you can easily distinguish the results.

In the second Visual Studio Test tasks configure the inverse filter:

Filter the second Visual Studio Test task to run the inverse set of tests.

Alternative filters

Test Categories are a very common way to filter tests, but you may have a large set of tests that already follow a different naming pattern. A common distinction is to put all Integration tests in their own assembly or in a different namespace.

This can be accomplished quite easily as well. To understand what filtering options you have available, here is an overview of what you can do.

Assembly name

The Test files follows the standard Azure Pipelines glob patterns:

• **/ recursive folders
• * wildcard search
• ? Single character placeholder

When your Integration tests are in their own projects you can use the project and assembly name to select these tests:

• **/*IntegrationTests.dll for integration tests
• **/*UnitTests.dll for unit tests

Namespace or class

The Test filter criteria option can filter on a number of items, Full name and Classname (not all projects support the class name filter) can be used to filter. You can either use an exact match (=), or a contains filter (~), for example:

• FullyQualifiedName~Integration
• ClassName~Important

The operators and properties are explained here quite well.

Category or Priority

As mentioned above you can use Test Categories, or you can filter on Priority.

• Priority=1

You can also combine multiple items using & (and) and | (or):

• Priority=1&TestCategory=Critical
• Priority=2&TestCategory=Critical
• Priority=3&TestCategory=Critical

This would queue 3 runs, each would run less important tests that are still critical, before running the rest of the tests.

Continue on test failure

Each of the test tasks can be configured to not fail the executing pipeline. This is a standard feature that's found on every task on Azure DevOps and Team Foundation Server 2017 or later.

I sometimes use this to filter out flaky tests:

Run your flaky tests in a separate run

Don't fail the pipeline

This way you do get statistics on these tests and can refactor them over time to be stable, yet still get releases out of the door. It's useful on code bases with technical debt and it can help while your working on cleaning-up the mess.

Source: StackOverflow