Azure DevOps Services | Azure DevOps Server 2022 - Azure DevOps Server 2019
Automate test cases in your test plans and run them directly from Azure Test Plans. Automated tests provide you with the following benefits:
In the Test Plans page, choose your test plan, open the shortcut menu, and then select Test plan settings.
In the Test plan settings dialog, select the build pipeline that generates builds that contain the test binaries (both, Classic and YAML build pipelines are supported). You can then select a specific build number to test, or let the system automatically use the latest build when tests are run.
You need a release pipeline that was created from the Run automated tests from Test Manager template to run tests from test plans in Azure Test Plans. If you have an existing release pipeline created using this template, select it and then select the existing stage in the release pipeline for the test execution (both, Classic and YAML release pipelines are supported). Otherwise, select Create new in the dialog to create a new release pipeline that contains a single stage with the Visual Studio Test task already added.
How do I pass parameters to my test code from a build or release pipeline?
Assign meaningful names to the release pipeline and stage as required.
If Visual Studio is already installed on the agent computer, skip this step. If not, add the Visual Studio Test Platform Installer task to the pipeline definition.
Add the Visual Studio Test task to the release pipeline and configure it as follows:
Verify that you are using the version 3 of the Visual Studio Test task.
Verify that Select tests using is set to Test run. What does this setting mean?
For the vsTestVersion, select toolsInstaller.
If you have UI tests that run on physical browsers or thick clients, ensure that the agent is set to run as an interactive process with autologon enabled. You must set up an agent to run interactively before you queue the build or release. The Test mix contains UI tests checkbox doesn't configure the agent in interactive mode automatically - used only as a reminder to configure the agent appropriately to avoid failures.
If you're running UI tests on a headless browser, the interactive process configuration isn't required.
Select how the test platform gets provisioned, and the version of Visual Studio or the location of the test platform that is installed on the test machines.
If your tests need input parameters such as app URLs or database connection strings, select the relevant settings file from the build artifacts. You can use the Publish build artifacts tasks in your build pipeline to publish the settings file in a drop location if this file isn't included in the artifacts. In the following example, the application URL is exposed in the run settings file, and is overridden to set it to a staging URL using the Override test run parameters setting.
For information about the option settings of the Visual Studio Test task, see Visual Studio Test task.
Choose the Agent job item and verify that the deployment queue is set to the one containing the machines where you want to run the tests. If your tests require special machines from the agent pool, you can add demands that select at runtime.
You might be able to minimize test times by distributing tests across multiple agents by setting Parallelism to Multiple executions and specifying the number of agents.
Note
If you're running UI tests such as CodeUI or Selenium on physical browsers such as IE, Firefox, or Chrome, the agent on the machines must be running in interactive mode and not as a service. More details.
In the Pipeline page of the release pipeline, verify that the build pipeline containing the test binaries links to this release pipeline as an artifact source.
Save the release pipeline.
If you chose Create new in the Test plan settings dialog in step 2 of this example, return to the browser page that contains your test plan settings. In the Test plan settings dialog, select the release pipeline and stage you saved.
In the Test Plans web portal, open the test plan and select a test suite that contains the automated tests.
Select the test case(s) you want to run, then click on Run for web application.
The test binaries for these tests must be available in the build artifacts generated by your build pipeline.
The system checks that only automated tests get selected (any manual tests are ignored), validates the stage to ensure the Visual Studio Test task is present and has valid settings, checks the user's permission to create a release for the selected release pipeline, creates a test run, and then triggers the creation of a release to the selected stage.
Select View test run to view the test progress and analyze the failed tests. Test results have the relevant information for debugging failed tests such as the error message, stack trace, console logs, and attachments.
After test execution is complete, the Runs page of the Azure Test Plans shows the test results. The Run summary page shows an overview of the run.
There's a link to the Release used to run the tests, which makes it easy to find the release that ran the tests if you need to come back later and analyze the results. Also use this link if you want to open the release to view the release logs.
Note
Manually attachment of files isn't supported for automated test results.
What are the typical error scenarios or issues I should look out for if my tests don't run?
The Test results page lists the results for each test in the test run. Select a test to see debugging information for failed tests such as the error message, stack trace, console logs, and attachments.
In Test Plans go to the Runs page where you can find an overiew of all your test runs. From here, you can open the detailed view of each test run.
See the following frequently asked questions (FAQs) about Azure Test Plans.
Q: What permissions do I need to run automated tests from Azure Test Plans?A: Be a Project Contributor, or have the following permissions:
For more information, see Release permissions.
Q: Can I override the build or stage set at the test plan level for a specific instance of test run?A: Yes, you can do this using the Run with options command. Open the shortcut menu for the test suite in the left column and select Run with options.
Enter the following values in the Run with options dialog and then select OK:
A: Azure Pipelines offers a compelling orchestration workflow to obtain test binaries as artifacts and run tests. This workflow shares the same concepts used in the scheduled testing workflow, meaning users running tests in scheduled workflow find it easy to adapt; for example, by cloning an existing scheduled testing release pipeline.
Another major benefit is the availability of a rich set of tasks in the task catalog that enable a range of activities to be performed before and after running tests. Examples include preparing and cleaning test data, creating and cleaning configuration files, and more.
Q: How does selecting "Test run" in the Visual Studio Test task version 2 work?A: The Test management subsystem uses the test run object to pass the list of tests selected for execution. The test task looks up the test run identifier, extracts the test execution information such as the container and test method names, runs the tests, updates the test run results, and sets the test points associated with the test results in the test run. From an auditing perspective, the Visual Studio task provides a trace from the historical releases and the test run identifiers to the tests that were submitted for on-demand test execution.
Q: Should the agent run in interactive mode or as a service?A: If you run UI tests such as coded UI or Selenium tests, the agent on the test machines must be running in interactive mode with autologon enabled, not as a service, to allow the agent to launch a web browser. If you're using a headless browser such as PhantomJS, the agent can be run as a service or in interactive mode. For more information, see Build and release agents, Deploy an agent on Windows, and Agent pools.
Q: Where can I find detailed documentation on how to run Selenium tests?A: See Get started with Selenium testing.
Q: What happens if I select multiple configurations for the same test?A: Currently, the on-demand workflow isn't configuration-aware.
Q: What if I need to download product binaries and test binaries from different builds? Or if I need to obtain artifacts from a source such as Jenkins?A: The current capability is optimized for a single team build to be tested on-demand using an Azure Pipelines workflow. We evaluate support for multi-artifact releases, including non-Azure Pipelines artifacts such as Jenkins, based on user feedback.
Q: I already have a scheduled testing release pipeline. Can I reuse the same pipeline to run test on-demand, or should I create a new pipeline?A: We recommend you use a separate release pipeline and stage for on-demand automated testing from Azure Test Plans because:
You might not want to deploy the app every time you want to run a few on-demand tests. Scheduled testing stages are typically set up to deploy the product and then run tests.
New releases are triggered for every on-demand run. If you have many testers who execute a few on-demand test runs every day, your scheduled testing release pipeline could be overloaded with releases for these runs, making it difficult to find releases that trigger for the pipeline that contains scheduled testing and deployment to production.
You might want to configure the Visual Studio Test task with a Test run identifier as an input so that you can trace what triggered the release. For more information, see How does selecting "Test run (for on-demand runs)" in the Visual Studio Test task work?.
A: No. Microsoft Test Manager doesn't support running automated tests against Team Foundation builds. It only works in the web-based interface for Azure Pipelines. All new manual and automated testing product development investments are in the web-based interface. No further development is planned for Microsoft Test Manager. See Guidance on Microsoft Test Manager usage.
Q: I have multiple testers in my team. Can they run tests from different test suites or test plans in parallel using the same release pipeline?A: They can use the same release pipeline to trigger multiple test runs in parallel if:
The agent pool associated with the stage has sufficient agents to cater for parallel requests. If sufficient agents aren't available, runs can still be triggered but releases queue for processing until agents become available.
You have sufficient jobs to enable parallel jobs. For more information, see Parallel jobs in Azure Pipelines or Parallel jobs in TFS.
Testers don't run the same tests in parallel. Doing so might cause results to be overwritten depending on the order of execution.
To enable multiple different test runs to execute in parallel, set the Azure Pipelines stage trigger option for behavior when multiple releases are waiting to be deployed as follows:
If your application supports tests running in parallel from different sources, set this option to Allow multiple releases to be deployed at the same time.
If your application doesn't support tests running in parallel from different sources, set this option to Allow only one active deployment at a time.
A: Use a runsettings file to pass values as parameters to your test code. For example, in a release that contains several stages, you can pass the appropriate app URL to each the test tasks in each one. The runsettings file and matching parameters must be specified in the Visual Studio Test task.
Q: What are the typical error scenarios or issues I should look out for if my tests don't run?A: Check and resolve issues as follows:
The release pipeline and stage in which I want to run tests aren't shown after I select the build.
I get an error that I don't have sufficient permission to trigger a release.
I get an error that no automated tests were found.
My tests didn't execute, and I suspect the release pipeline is incorrect.
My tests go into the error state, or remain "in-progress" even after release to the stage is triggered.
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4