未加星标

Staged execution of tests in Azure DevOps Pipelines

字体大小 | |
[系统(windows) 所属分类 系统(windows) | 发布者 店小二03 | 时间 2019 | 作者 红领巾 ] 0人收藏点击收藏

When running Unit Tests in your build system, you may want to first run the most important suite, the one that should never fail, the ones that are currently being changed, finally the regression suite and integration tests that may be slower to execute.

The Visual Studio Test task supports this feature, though many probably never knew that this was in the system since the first release of the Workflow based Team Build 2010.

In your build definition you can easily add multiple test runs, and for each test run define which tests should run. Not just fixed list of tests you have to curate, but a dynamic list based on attributes of the test.

The following steps will take you through the basic gist to get this working in your Azure Pipeline.

1. Indentify the tests in different categories Go through your test base and apply categories to your automated tests. Use the [TestCategory] attribute for MsTest and the [Category] attribute for NUnit and [Trait] for xUnit. // MsTest [TestCategory("Critical")] [TestMethod] public void MyCriticalTest {} // NUnit [Category("Critical")] [Test] public void MyCriticalTest {} // xUnit [Trait("Category", "Critical")] [Fact] public void MyCriticalTest {}

The Test Filter options for Visual Studio Test can also filter on other characteristics of your test, such as the name of the test, it's parent class, namespace or assembly. You can combine these elements:

// MsTest [TestCategory("Critical")] [TestPriority(1)] [TestMethod] public void MyCriticalTest {} 2. Create multiple test runs in Team Build

Once you've categorized your tests create the test runs in Azure Pipelines. Each stage or set of tests will map to its own test run. In this example we're going to distinguish critical tests from non-critical tests using a simple attribute.

In your Azure Pipeline add multiple Visual Studio Test tasks, one for each set of tests you want to run:


Staged execution of tests in Azure DevOps Pipelines
Add a Visual Studio Test task for each set of tests to run.

We're going to configure the first run to execute the critical tests (and fail the build when any of these tests fail). Only when these tests succeed will we run the non-critical tests.

In the VsTest - Critical Tests configure the test filter to only run Critical tests:


Staged execution of tests in Azure DevOps Pipelines
Set the Test filter Criteria to TestCategory=Critical
Staged execution of tests in Azure DevOps Pipelines
Name the test run so you can easily distinguish the results.

In the second Visual Studio Test tasks configure the inverse filter:


Staged execution of tests in Azure DevOps Pipelines
Filter the second Visual Studio Test task to run the inverse set of tests. Alternative filters

Test Categories are a very common way to filter tests, but you may have a large set of tests that already follow a different naming pattern. A common distinction is to put all Integration tests in their own assembly or in a different namespace.

This can be accomplished quite easily as well. To understand what filtering options you have available, here is an overview of what you can do.

Assembly name

The Test files follows the standard Azure Pipelines glob patterns :

**/ * ?

When your Integration tests are in their own projects you can use the project and assembly name to select these tests:

**/*IntegrationTests.dll **/*UnitTests.dll

Namespace or class

The Test filter criteria option can filter on a number of items, Full name and Classname (not all projects support the class name filter) can be used to filter. You can either use an exact match ( = ), or a contains filter ( ~ ), for example:

FullyQualifiedName~Integration ClassName~Important

The operators and properties are explained here quite well.

Category or Priority

As mentioned above you can use Test Categories, or you can filter on Priority.

Priority=1

You can also combine multiple items using & (and) and | (or):

Priority=1&TestCategory=Critical Priority=2&TestCategory=Critical Priority=3&TestCategory=Critical

This would queue 3 runs, each would run less important tests that are still critical, before running the rest of the tests.

Continue on test failure

Each of the test tasks can be configured to not fail the executing pipeline. This is a standard feature that's found on every task on Azure DevOps and Team Foundation Server 2017 or later.

I sometimes use this to filter out flaky tests:


Staged execution of tests in Azure DevOps Pipelines
Run your flaky tests in a separate run
Staged execution of tests in Azure DevOps Pipelines
Don't fail the pipeline

This way you do get statistics on these tests and can refactor them over time to be stable, yet still get releases out of the door. It's useful on code bases with technical debt and it can help while your working on cleaning-up the mess.

Source: StackOverflow

本文系统(windows)相关术语:三级网络技术 计算机三级网络技术 网络技术基础 计算机网络技术

代码区博客精选文章
分页:12
转载请注明
本文标题:Staged execution of tests in Azure DevOps Pipelines
本站链接:https://www.codesec.net/view/627828.html


1.凡CodeSecTeam转载的文章,均出自其它媒体或其他官网介绍,目的在于传递更多的信息,并不代表本站赞同其观点和其真实性负责;
2.转载的文章仅代表原创作者观点,与本站无关。其原创性以及文中陈述文字和内容未经本站证实,本站对该文以及其中全部或者部分内容、文字的真实性、完整性、及时性,不作出任何保证或承若;
3.如本站转载稿涉及版权等问题,请作者及时联系本站,我们会及时处理。
登录后可拥有收藏文章、关注作者等权限...
技术大类 技术大类 | 系统(windows) | 评论(0) | 阅读(69)