A RetroSearch Logo

Home - News ( United States | United Kingdom | Italy | Germany ) - Football scores

Search Query:

Showing content from https://learn.microsoft.com/en-us/azure/devops/pipelines/repos/github below:

Build GitHub repositories - Azure Pipelines

Azure DevOps Services

Azure Pipelines can automatically build and validate every pull request and commit to your GitHub repository. This article describes how to configure the integration between GitHub and Azure Pipelines.

If you're new to pipelines integration with GitHub, follow the steps in Create your first pipeline. Come back to this article to learn more about configuring and customizing the integration between GitHub and Azure Pipelines.

Organizations and users

GitHub and Azure Pipelines are two independent services that integrate well together. Each of them have their own organization and user management. This section makes a recommendation on how to replicate the organization and users from GitHub to Azure Pipelines.

Organizations

GitHub's structure consists of organizations and user accounts that contain repositories. See GitHub's documentation.

Azure DevOps' structure consists of organizations that contain projects. See Plan your organizational structure.

Azure DevOps can reflect your GitHub structure with:

To set up an identical structure in Azure DevOps:

  1. Create a DevOps organization named after your GitHub organization or user account. It will have a URL like https://dev.azure.com/your-organization.
  2. In the DevOps organization, create projects named after your repositories. They’ll have URLs like https://dev.azure.com/your-organization/your-repository.
  3. In the DevOps Project, create pipelines named after the GitHub organization and repository they build, such as your-organization.your-repository. Then, it's clear which repositories they're for.

Following this pattern, your GitHub repositories and Azure DevOps Projects will have matching URL paths. For example:

Service URL GitHub https://github.com/python/cpython Azure DevOps https://dev.azure.com/python/cpython Users

Your GitHub users don’t automatically get access to Azure Pipelines. Azure Pipelines is unaware of GitHub identities. For this reason, there’s no way to configure Azure Pipelines to automatically notify users of a build failure or a PR validation failure using their GitHub identity and email address. You must explicitly create new users in Azure Pipelines to replicate GitHub users. Once you create new users, you can configure their permissions in Azure DevOps to reflect their permissions in GitHub. You can also configure notifications in DevOps using their DevOps identity.

GitHub organization roles

GitHub organization member roles are found at https://github.com/orgs/your-organization/people (replace your-organization).

DevOps organization member permissions are found at https://dev.azure.com/your-organization/_settings/security (replace your-organization).

Roles in a GitHub organization and equivalent roles in an Azure DevOps organization are shown below.

GitHub organization role DevOps organization equivalent Owner Member of Project Collection Administrators Billing manager Member of Project Collection Administrators Member Member of Project Collection Valid Users. By default, the Member group lacks permission to create new projects. To change the permission, set the group's Create new projects permission to Allow, or create a new group with permissions you need. GitHub user account roles

A GitHub user account has one role, which is ownership of the account.

DevOps organization member permissions are found at https://dev.azure.com/your-organization/_settings/security (replace your-organization).

The GitHub user account role maps to DevOps organization permissions as follows.

GitHub user account role DevOps organization equivalent Owner Member of Project Collection Administrators GitHub repository permissions

GitHub repository permissions are found at https://github.com/your-organization/your-repository/settings/collaboration (replace your-organization and your-repository).

DevOps project permissions are found at https://dev.azure.com/your-organization/your-project/_settings/security (replace your-organization and your-project).

Equivalent permissions between GitHub repositories and Azure DevOps Projects are as follows.

GitHub repository permission DevOps project equivalent Admin Member of Project Administrators Write Member of Contributors Read Member of Readers

If your GitHub repository grants permission to teams, you can create matching teams in the Teams section of your Azure DevOps project settings. Then, add the teams to the security groups above, just like users.

Pipeline-specific permissions

To grant permissions to users or teams for specific pipelines in a DevOps project, follow these steps:

  1. Visit the project's Pipelines page (for example, https://dev.azure.com/your-organization/your-project/_build).
  2. Select the pipeline for which to set specific permissions.
  3. From the '...' context menu, select Security.
  4. Select Add... to add a specific user, team, or group and customize their permissions for the pipeline.
Access to GitHub repositories

You create a new pipeline by first selecting a GitHub repository and then a YAML file in that repository. The repository in which the YAML file is present is called self repository. By default, this is the repository that your pipeline builds.

You can later configure your pipeline to check out a different repository or multiple repositories. To learn how to do this, see multi-repo checkout.

You create a new pipeline by first selecting GitHub for repository type, and then one of the repositories you have access to.

Azure Pipelines must be granted access to your repositories to trigger their builds, and fetch their code during builds.

There are three authentication types for granting Azure Pipelines access to your GitHub repositories while creating a pipeline.

GitHub app authentication

The Azure Pipelines GitHub App is the recommended authentication type for continuous integration pipelines. After you install the GitHub App in your GitHub account or organization, your pipeline will run without using your personal GitHub identity. Builds and GitHub status updates will be performed using the Azure Pipelines identity. The app works with GitHub Checks to display build, test, and code coverage results in GitHub.

To use the GitHub App, install it in your GitHub organization or user account for some or all repositories. The GitHub App can be installed and uninstalled from the app's homepage.

After installation, the GitHub App will become Azure Pipelines' default method of authentication to GitHub (instead of OAuth) when pipelines are created for the repositories.

If you install the GitHub App for all repositories in a GitHub organization, you don't need to worry about Azure Pipelines sending mass emails or automatically setting up pipelines on your behalf. However, if the app is installed for all repositories, the token used by the application will have access to all repositories, including private ones. For security reasons, it is recommended to separate private and public repositories at the organization level. This means having a dedicated organization only for public projects without private repositories. If, for some reason, there is a need to have public and private repositories in the same organization, instead of using access for all repositories, explicitly select the repositories to which the application should have access. This requires more work for admins but ensures better security management.

Permissions needed in GitHub

Installation of Azure Pipelines GitHub app requires you to be a GitHub organization owner or repository admin. In addition, to create a pipeline for a GitHub repository with continuous integration and pull request triggers, you must have the required GitHub permissions configured. Otherwise, the repository will not appear in the repository list while creating a pipeline. Depending on the authentication type and ownership of the repository, ensure that the appropriate access is configured.

GitHub App permissions

The GitHub App requests the following permissions during installation:

Permission How Azure Pipelines uses the permission Write access to code Only upon your deliberate action, Azure Pipelines will simplify creating a pipeline by committing a YAML file to a selected branch of your GitHub repository. Read access to metadata Azure Pipelines will retrieve GitHub metadata for displaying the repository, branches, and issues associated with a build in the build's summary. Read and write access to checks Azure Pipelines will read and write its own build, test, and code coverage results to be displayed in GitHub. Read and write access to pull requests Only upon your deliberate action, Azure Pipelines will simplify creating a pipeline by creating a pull request for a YAML file that was committed to a selected branch of your GitHub repository. Pipelines retrieves request metadata to display in build summaries associated with pull requests. Troubleshooting GitHub App installation

GitHub may display an error such as:

You do not have permission to modify this app on your-organization. Please contact an Organization Owner.

This means that the GitHub App is likely already installed for your organization. When you create a pipeline for a repository in the organization, the GitHub App will automatically be used to connect to GitHub.

Create pipelines in multiple Azure DevOps organizations and projects

Once the GitHub App is installed, pipelines can be created for the organization's repositories in different Azure DevOps organizations and projects. However, if you create pipelines for a single repository in multiple Azure DevOps organizations, only the first organization's pipelines can be automatically triggered by GitHub commits or pull requests. Manual or scheduled builds are still possible in secondary Azure DevOps organizations.

OAuth authentication

OAuth is the simplest authentication type to get started with for repositories in your personal GitHub account. GitHub status updates will be performed on behalf of your personal GitHub identity. For pipelines to keep working, your repository access must remain active. Some GitHub features, like Checks, are unavailable with OAuth and require the GitHub App.

To use OAuth, select Choose a different connection below the list of repositories while creating a pipeline. Then, select Authorize to sign into GitHub and authorize with OAuth. An OAuth connection will be saved in your Azure DevOps project for later use, and used in the pipeline being created.

Permissions needed in GitHub

To create a pipeline for a GitHub repository with continuous integration and pull request triggers, you must have the required GitHub permissions configured. Otherwise, the repository will not appear in the repository list while creating a pipeline. Depending on the authentication type and ownership of the repository, ensure that the appropriate access is configured.

Revoke OAuth access

After authorizing Azure Pipelines to use OAuth, to later revoke it and prevent further use, visit OAuth Apps in your GitHub settings. You can also delete it from the list of GitHub service connections in your Azure DevOps project settings.

Personal access token (PAT) authentication

PATs are effectively the same as OAuth, but allow you to control which permissions are granted to Azure Pipelines. Builds and GitHub status updates will be performed on behalf of your personal GitHub identity. For builds to keep working, your repository access must remain active.

To create a PAT, visit Personal access tokens in your GitHub settings. The required permissions are repo, admin:repo_hook, read:user, and user:email. These are the same permissions required when using OAuth above. Copy the generated PAT to the clipboard and paste it into a new GitHub service connection in your Azure DevOps project settings. For future recall, name the service connection after your GitHub username. It will be available in your Azure DevOps project for later use when creating pipelines.

Permissions needed in GitHub

To create a pipeline for a GitHub repository with continuous integration and pull request triggers, you must have the required GitHub permissions configured. Otherwise, the repository will not appear in the repository list while creating a pipeline. Depending on the authentication type and ownership of the repository, ensure that the following access is configured.

Revoke PAT access

After authorizing Azure Pipelines to use a PAT, to later delete it and prevent further use, visit Personal access tokens in your GitHub settings. You can also delete it from the list of GitHub service connections in your Azure DevOps project settings.

CI triggers

Continuous integration (CI) triggers cause a pipeline to run whenever you push an update to the specified branches or you push specified tags.

YAML pipelines are configured by default with a CI trigger on all branches, unless the Disable implied YAML CI trigger setting, introduced in Azure DevOps sprint 227, is enabled. The Disable implied YAML CI trigger setting can be configured at the organization level or at the project level. When the Disable implied YAML CI trigger setting is enabled, CI triggers for YAML pipelines are not enabled if the YAML pipeline doesn't have a trigger section. By default, Disable implied YAML CI trigger is not enabled.

Branches

You can control which branches get CI triggers with a simple syntax:

trigger:
- main
- releases/*

You can specify the full name of the branch (for example, main) or a wildcard (for example, releases/*). See Wildcards for information on the wildcard syntax.

Note

You cannot use variables in triggers, as variables are evaluated at runtime (after the trigger has fired).

Note

If you use templates to author YAML files, then you can only specify triggers in the main YAML file for the pipeline. You cannot specify triggers in the template files.

For more complex triggers that use exclude or batch, you must use the full syntax as shown in the following example.

# specific branch build
trigger:
  branches:
    include:
    - main
    - releases/*
    exclude:
    - releases/old*

In the above example, the pipeline will be triggered if a change is pushed to main or to any releases branch. However, it won't be triggered if a change is made to a releases branch that starts with old.

If you specify an exclude clause without an include clause, then it is equivalent to specifying * in the include clause.

In addition to specifying branch names in the branches lists, you can also configure triggers based on tags by using the following format:

trigger:
  branches:
    include:
      - refs/tags/{tagname}
    exclude:
      - refs/tags/{othertagname}

If you didn't specify any triggers, and the Disable implied YAML CI trigger setting is not enabled, the default is as if you wrote:

trigger:
  branches:
    include:
    - '*'  # must quote since "*" is a YAML reserved character; we want a string

Important

When you specify a trigger, it replaces the default implicit trigger, and only pushes to branches that are explicitly configured to be included will trigger a pipeline. Includes are processed first, and then excludes are removed from that list.

Batching CI runs

If you have many team members uploading changes often, you may want to reduce the number of runs you start. If you set batch to true, when a pipeline is running, the system waits until the run is completed, then starts another run with all changes that have not yet been built.

# specific branch build with batching
trigger:
  batch: true
  branches:
    include:
    - main

Note

batch is not supported in repository resource triggers.

To clarify this example, let us say that a push A to main caused the above pipeline to run. While that pipeline is running, additional pushes B and C occur into the repository. These updates do not start new independent runs immediately. But after the first run is completed, all pushes until that point of time are batched together and a new run is started.

Note

If the pipeline has multiple jobs and stages, then the first run should still reach a terminal state by completing or skipping all its jobs and stages before the second run can start. For this reason, you must exercise caution when using this feature in a pipeline with multiple stages or approvals. If you wish to batch your builds in such cases, it is recommended that you split your CI/CD process into two pipelines - one for build (with batching) and one for deployments.

Paths

You can specify file paths to include or exclude.

# specific path build
trigger:
  branches:
    include:
    - main
    - releases/*
  paths:
    include:
    - docs
    exclude:
    - docs/README.md

When you specify paths, you must explicitly specify branches to trigger on if you are using Azure DevOps Server 2019.1 or lower. You can't trigger a pipeline with only a path filter; you must also have a branch filter, and the changed files that match the path filter must be from a branch that matches the branch filter. If you are using Azure DevOps Server 2020 or newer, you can omit branches to filter on all branches in conjunction with the path filter.

Wildcards are supported for path filters. For instance, you can include all paths that match src/app/**/myapp*. You can use wildcard characters (**, *, or ?) when specifying path filters.

Tags

In addition to specifying tags in the branches lists as covered in the previous section, you can directly specify tags to include or exclude:

# specific tag
trigger:
  tags:
    include:
    - v2.*
    exclude:
    - v2.0

If you don't specify any tag triggers, then by default, tags will not trigger pipelines.

Important

If you specify tags in combination with branch filters, the trigger will fire if either the branch filter is satisfied or the tag filter is satisfied. For example, if a pushed tag satisfies the branch filter, the pipeline triggers even if the tag is excluded by the tag filter, because the push satisfied the branch filter.

Opting out of CI Disabling the CI trigger

You can opt out of CI triggers entirely by specifying trigger: none.

# A pipeline with no CI trigger
trigger: none

Important

When you push a change to a branch, the YAML file in that branch is evaluated to determine if a CI run should be started.

Select Enable continuous integration on the Triggers tab to enable this trigger if you want the build to run whenever someone checks in code.

Batch changes

Select this check box if you have many team members uploading changes often and you want to reduce the number of builds you are running. If you select this option, when a build is running, the system waits until the run is completed and then queues another run of all changes that have not yet been built.

You can batch changes and build them together.

Note

If you use batching with a multi-stage YAML pipeline, then a run must reach a terminal state before the next one can start. This is often not desirable as a multi-stage pipeline may go through approvals and long-running deployment stages. In these cases, it is recommended that you follow one of these solutions:

Branch filters

You can specify the branches where you want to trigger builds. If you want to use wildcard characters, then type the branch specification (for example, features/modules/*) and then press Enter.

Path filters

If your Git repo is in Azure Repos or TFS, you can also specify path filters to reduce the set of files that you want to trigger a build.

Tips:

Example

For example, you want your build to be triggered by changes in main and most, but not all, of your feature branches. You also don't want builds to be triggered by changes to files in the tools folder.

Skipping CI for individual commits

You can also tell Azure Pipelines to skip running a pipeline that a push would normally trigger. Just include [skip ci] in the message or description of any of the commits that are part of a push, and Azure Pipelines will skip running CI for this push. You can also use any of the following variations.

Using the trigger type in conditions

It is a common scenario to run different steps, jobs, or stages in your pipeline depending on the type of trigger that started the run. You can do this using the system variable Build.Reason. For example, add the following condition to your step, job, or stage to exclude it from PR validations.

condition: and(succeeded(), ne(variables['Build.Reason'], 'PullRequest'))

Behavior of triggers when new branches are pushed

It is common to configure multiple pipelines for the same repository. For instance, you may have one pipeline to build the docs for your app and another to build the source code. You may configure CI triggers with appropriate branch filters and path filters in each of these pipelines. For instance, you may want one pipeline to trigger when you push an update to the docs folder, and another one to trigger when you push an update to your application code. In these cases, you need to understand how the pipelines are triggered when a new branch is created.

Here is the behavior when you push a new branch (that matches the branch filters) to your repository:

Wildcards

When specifying a branch, tag, or path, you may use an exact name or a wildcard. Wildcards patterns allow * to match zero or more characters and ? to match a single character.

trigger:
  branches:
    include:
    - main
    - releases/*
    - feature/*
    exclude:
    - releases/old*
    - feature/*-working
  paths:
    include:
    - docs/*.md
PR triggers

Pull request (PR) triggers cause a pipeline to run whenever a pull request is opened with one of the specified target branches, or when updates are made to such a pull request.

Branches

You can specify the target branches when validating your pull requests. For example, to validate pull requests that target main and releases/*, you can use the following pr trigger.

pr:
- main
- releases/*

This configuration starts a new run the first time a new pull request is created, and after every update made to the pull request.

You can specify the full name of the branch (for example, main) or a wildcard (for example, releases/*).

Note

You cannot use variables in triggers, as variables are evaluated at runtime (after the trigger has fired).

Note

If you use templates to author YAML files, then you can only specify triggers in the main YAML file for the pipeline. You cannot specify triggers in the template files.

GitHub creates a new ref when a pull request is created. The ref points to a merge commit, which is the merged code between the source and target branches of the pull request. The PR validation pipeline builds the commit that this ref points to. This means that the YAML file that is used to run the pipeline is also a merge between the source and the target branch. As a result, the changes you make to the YAML file in source branch of the pull request can override the behavior defined by the YAML file in target branch.

If no pr triggers appear in your YAML file, pull request validations are automatically enabled for all branches, as if you wrote the following pr trigger. This configuration triggers a build when any pull request is created, and when commits come into the source branch of any active pull request.

pr:
  branches:
    include:
    - '*'  # must quote since "*" is a YAML reserved character; we want a string

Important

When you specify a pr trigger with a subset of branches, a pipeline is triggered only when updates are pushed to those branches.

For more complex triggers that need to exclude certain branches, you must use the full syntax as shown in the following example. In this example, pull requests are validated that target main or releases/* and the branch releases/old* is excluded.

# specific branch
pr:
  branches:
    include:
    - main
    - releases/*
    exclude:
    - releases/old*
Paths

You can specify file paths to include or exclude. For example:

# specific path
pr:
  branches:
    include:
    - main
    - releases/*
  paths:
    include:
    - docs
    exclude:
    - docs/README.md

Tips:

Multiple PR updates

You can specify whether more updates to a PR should cancel in-progress validation runs for the same PR. The default is true.

# auto cancel false
pr:
  autoCancel: false
  branches:
    include:
    - main
Draft PR validation

By default, pull request triggers fire on draft pull requests and pull requests that are ready for review. To disable pull request triggers for draft pull requests, set the drafts property to false.

pr:
  autoCancel: boolean # indicates whether additional pushes to a PR should cancel in-progress runs for the same PR. Defaults to true
  branches:
    include: [ string ] # branch names which will trigger a build
    exclude: [ string ] # branch names which will not
  paths:
    include: [ string ] # file paths which must match to trigger a build
    exclude: [ string ] # file paths which will not trigger a build
  drafts: boolean # whether to build draft PRs, defaults to true
Opting out of PR validation

You can opt out of pull request validation entirely by specifying pr: none.

# no PR triggers
pr: none

For more information, see PR trigger in the YAML schema.

Note

If your pr trigger isn't firing, follow the troubleshooting steps in the FAQ.

Select the Pull request validation trigger and check the Enable pull request validation check box to enable builds on pull requests.

You can specify branches to include and exclude. Select a branch name from the drop-down menu and select Include or Exclude as appropriate. For included branches, a build will be triggered on each push to a pull request targeting that branch.

If you have an open PR and you push changes to its source branch, multiple pipelines may run:

Finally, after you merge the PR, Azure Pipelines will run the CI pipelines triggered by pushes to the target branch, if the merge commit's message or description doesn't contain [skip ci] (or any of its variants).

Protected branches

You can run a validation build with each commit or pull request that targets a branch, and even prevent pull requests from merging until a validation build succeeds.

To configure mandatory validation builds for a GitHub repository, you must be its owner, a collaborator with the Admin role, or a GitHub organization member with the Write role.

  1. First, create a pipeline for the repository and build it at least once so that its status is posted to GitHub, thereby making GitHub aware of the pipeline's name.

  2. Next, follow GitHub's documentation for configuring protected branches in the repository's settings.

    For the status check, select the name of your pipeline in the Status checks list.

Important

If your pipeline doesn't show up in this list, please ensure the following:

Contributions from external sources

If your GitHub repository is open source, you can make your Azure DevOps project public so that anyone can view your pipeline's build results, logs, and test results without signing in. When users outside your organization fork your repository and submit pull requests, they can view the status of builds that automatically validate those pull requests.

You should keep in mind the following considerations when using Azure Pipelines in a public project when accepting contributions from external sources.

Access restrictions

Be aware of the following access restrictions when you're running pipelines in Azure DevOps public projects:

Contributions from forks

Important

These settings affect the security of your pipeline.

When you create a pipeline, it’s automatically triggered for pull requests from forks of your repository. You can change this behavior, carefully considering how it affects security. To enable or disable this behavior:

  1. Go to your Azure DevOps project. Select Pipelines, locate your pipeline, and select Edit.
  2. Select the Triggers tab. After enabling the Pull request trigger, enable or disable the Build pull requests from forks of this repository check box.

By default with GitHub pipelines, secrets associated with your build pipeline aren’t made available to pull request builds of forks. These secrets are enabled by default with GitHub Enterprise Server pipelines. Secrets include:

To bypass this precaution on GitHub pipelines, enable the Make secrets available to builds of forks check box. Be aware of this setting's effect on security.

Note

When you enable fork builds to access secrets, Azure Pipelines by default restricts the access token used for fork builds. It has more limited access to open resources than a normal access token. To give fork builds the same permissions as regular builds, enable the Make fork builds have the same permissions as regular builds setting.

For more information, see Repository protection - Forks.

You can define centrally how pipelines build PRs from forked GitHub repositories using the Limit building pull requests from forked GitHub repositories control. It's available at organization and project level. You can choose to:

Starting with Sprint 229, to improve the security of your pipelines, Azure Pipelines no longer automatically builds pull requests from forked GitHub repositories. For new projects and organizations, the default value of the Limit building pull requests from forked GitHub repositories setting is Disable building pull requests from forked repositories.

When you choose the Securely build pull requests from forked repositories option, all pipelines, organization or project-wide, cannot make secrets available to builds of PRs from forked repositories, cannot make these builds have the same permissions as normal builds, and must be triggered by a PR comment. Projects can still decide to not allow pipelines to build such PRs.

When you choose the Customize option, you can define how to restrict pipeline settings. For example, you can ensure that all pipelines require a comment in order to build a PR from a forked GitHub repo, when the PR belongs to non-team members and non-contributors. But, you can choose to allow them to make secrets available to such builds. Projects can decide to not allow pipelines to build such PRs, or to build them securely, or have even more restrictive settings than what is specified at the organization level.

The control is off for existing organizations. Starting September 2023, new organizations have Securely build pull requests from forked repositories turned on by default.

Important security considerations

A GitHub user can fork your repository, change it, and create a pull request to propose changes to your repository. This pull request could contain malicious code to run as part of your triggered build. Such code can cause harm in the following ways:

Repository collaborators can comment on a pull request to manually run a pipeline. Here are a few common reasons for why you might want to do this:

To enable comment triggers, you must follow the following two steps:

  1. Enable pull request triggers for your pipeline, and make sure that you didn’t exclude the target branch.
  2. In the Azure Pipelines web portal, edit your pipeline and choose More actions, Triggers. Then, under Pull request validation, enable Require a team member's comment before building a pull request.

With these two changes, the pull request validation build won’t be triggered automatically, unless Only on pull requests from non-team members is selected and the PR is made by a team member. Only repository owners and collaborators with 'Write' permission can trigger the build by commenting on the pull request with /AzurePipelines run or /AzurePipelines run <pipeline-name>.

The following commands can be issued to Azure Pipelines in comments:

Command Result /AzurePipelines help Display help for all supported commands. /AzurePipelines help <command-name> Display help for the specified command. /AzurePipelines run Run all pipelines that are associated with this repository and whose triggers don’t exclude this pull request. /AzurePipelines run <pipeline-name> Run the specified pipeline unless its triggers exclude this pull request.

Note

For brevity, you can comment using /azp instead of /AzurePipelines.

Important

Responses to these commands will appear in the pull request discussion only if your pipeline uses the Azure Pipelines GitHub App.

If you have the necessary repository permissions, but pipelines aren't getting triggered by your comments, make sure that your membership is public in the repository's organization, or directly add yourself as a repository collaborator. Pipelines can’t see private organization members unless they are direct collaborators or belong to a team that is a direct collaborator. You can change your GitHub organization membership from private to public here (replace Your-Organization with your organization name): https://github.com/orgs/Your-Organization/people.

Informational runs

An informational run tells you Azure DevOps failed to retrieve a YAML pipeline's source code. Source code retrieval happens in response to external events, for example, a pushed commit. It also happens in response to internal triggers, for example, to check if there are code changes and start a scheduled run or not. Source code retrieval can fail for multiple reasons, with a frequent one being request throttling by the git repository provider. The existence of an informational run doesn't necessarily mean Azure DevOps was going to run the pipeline.

An informational run looks like in the following screenshot.

You can recognize an informational run by the following attributes:

Learn more about informational runs.

Checkout

When a pipeline is triggered, Azure Pipelines pulls your source code from the Azure Repos Git repository. You can control various aspects of how your code is checked out.

Note

When you include a checkout step in your pipeline, we run the following command: git -c fetch --force --tags --prune --prune-tags --progress --no-recurse-submodules origin --depth=1. If this default doesn't meet your needs, you can choose to exclude built-in checkout by checkout: none and then use a script task to perform your own checkout.

Preferred version of Git

The Windows agent comes with its own copy of Git. If you prefer to supply your own Git rather than use the included copy, set System.PreferGitFromPath to true. The System.PreferGitFromPath setting is always true on non-Windows agents.

Checkout path

If you're checking out a single repository, by default, your source code is checked out into a directory called s. For YAML pipelines, you can change this by specifying checkout with a path. The specified path is relative to $(Agent.BuildDirectory). For example: if the checkout path value is mycustompath and $(Agent.BuildDirectory) is C:\agent\_work\1, then the source code is checked out into C:\agent\_work\1\mycustompath.

If you're using multiple checkout steps and checking out multiple repositories, and not explicitly specifying the folder using path, each repository is placed in a subfolder of s named after the repository. For example if you check out two repositories named tools and code, the source code is checked out into C:\agent\_work\1\s\tools and C:\agent\_work\1\s\code.

Please note that the checkout path value can't be set to go up any directory levels above $(Agent.BuildDirectory), so path\..\anotherpath will result in a valid checkout path (i.e. C:\agent\_work\1\anotherpath), but a value like ..\invalidpath will not (i.e. C:\agent\_work\invalidpath).

You can configure the path setting in the Checkout step of your pipeline.

steps:
- checkout: self  # self represents the repo where the initial Pipelines YAML file was found
  clean: boolean  # whether to fetch clean each time
  fetchDepth: number  # the depth of commits to ask Git to fetch
  lfs: boolean  # whether to download Git-LFS files
  submodules: true | recursive  # set to 'true' for a single level of submodules or 'recursive' to get submodules of submodules
  path: string  # path to check out source code, relative to the agent's build directory (e.g. \_work\1)
  persistCredentials: boolean  # set to 'true' to leave the OAuth token in the Git config after the initial fetch

The path setting isn't configurable in the classic editor. Your source code is checked out into a directory called s, which is relative to $(Agent.BuildDirectory). For example: if $(Agent.BuildDirectory) is C:\agent\_work\1, then the source code is checked out into C:\agent\_work\1\mycustompath.

Submodules

You can configure the submodules setting in the Checkout step of your pipeline if you want to download files from submodules.

steps:
- checkout: self  # self represents the repo where the initial Pipelines YAML file was found
  clean: boolean  # whether to fetch clean each time
  fetchDepth: number  # the depth of commits to ask Git to fetch
  lfs: boolean  # whether to download Git-LFS files
  submodules: true | recursive  # set to 'true' for a single level of submodules or 'recursive' to get submodules of submodules
  path: string  # path to check out source code, relative to the agent's build directory (e.g. \_work\1)
  persistCredentials: boolean  # set to 'true' to leave the OAuth token in the Git config after the initial fetch

You can configure the Submodules setting from the properties of the Get sources task in your pipeline if you want to download files from submodules.

The build pipeline will check out your Git submodules as long as they are:

Alternative to using the Checkout submodules option

In some cases you can't use the Checkout submodules option. You might have a scenario where a different set of credentials are needed to access the submodules. This can happen, for example, if your main repository and submodule repositories aren't stored in the same Azure DevOps organization, or if your job access token doesn't have access to the repository in a different project.

If you can't use the Checkout submodules option, then you can instead use a custom script step to fetch submodules. First, get a personal access token (PAT) and prefix it with pat:. Next, base64-encode this prefixed string to create a basic auth token. Finally, add this script to your pipeline:

git -c http.https://<url of submodule repository>.extraheader="AUTHORIZATION: Basic <BASE64_ENCODED_STRING>" submodule update --init --recursive

Be sure to replace "<BASE64_ENCODED_STRING>" with your Base64-encoded "pat:token" string.

Use a secret variable in your project or build pipeline to store the basic auth token that you generated. Use that variable to populate the secret in the above Git command.

Note

Q: Why can't I use a Git credential manager on the agent? A: Storing the submodule credentials in a Git credential manager installed on your private build agent is usually not effective as the credential manager may prompt you to re-enter the credentials whenever the submodule is updated. This isn't desirable during automated builds when user interaction isn't possible.

Sync tags

Important

The sync tags feature is supported in Azure Repos Git with Azure DevOps Server 2022.1 and higher.

The checkout step uses the --tags option when fetching the contents of a Git repository. This causes the server to fetch all tags as well as all objects that are pointed to by those tags. This increases the time to run the task in a pipeline, particularly if you have a large repository with a number of tags. Furthermore, the checkout step syncs tags even when you enable the shallow fetch option, thereby possibly defeating its purpose. To reduce the amount of data fetched or pulled from a Git repository, Microsoft has added a new option to check out to control the behavior of syncing tags. This option is available both in classic and YAML pipelines.

Whether to synchronize tags when checking out a repository can be configured in YAML by setting the fetchTags property, and in the UI by configuring the Sync tags setting.

You can configure the fetchTags setting in the Checkout step of your pipeline.

To configure the setting in YAML, set the fetchTags property.

steps:
- checkout: self
  fetchTags: true

You can also configure this setting by using the Sync tags option in the pipeline settings UI.

  1. Edit your YAML pipeline and choose More actions, Triggers.

  2. Choose YAML, Get sources.

  3. Configure the Sync tags setting.

Note

If you explicitly set fetchTags in your checkout step, that setting takes priority over the setting configured in the pipeline settings UI.

You can configure the Sync tags setting from the properties of the Get sources task in your pipeline.

Default behavior

Note

If you explicitly set fetchTags in your checkout step, that setting takes priority over the setting configured in the pipeline settings UI.

Shallow fetch

You may want to limit how far back in history to download. Effectively this results in git fetch --depth=n. If your repository is large, this option might make your build pipeline more efficient. Your repository might be large if it has been in use for a long time and has sizeable history. It also might be large if you added and later deleted large files.

Note

In some organizations, new pipelines created after the September 2022 Azure DevOps sprint 209 update have Shallow fetch enabled by default and configured with a depth of 1. Previously the default wasn't to shallow fetch.

To check your pipeline, view the Shallow fetch setting in the pipeline settings UI as described in the following section.

You can configure the fetchDepth setting in the Checkout step of your pipeline.

steps:
- checkout: self  # self represents the repo where the initial Pipelines YAML file was found
  clean: boolean  # whether to fetch clean each time
  fetchDepth: number  # the depth of commits to ask Git to fetch
  lfs: boolean  # whether to download Git-LFS files
  submodules: true | recursive  # set to 'true' for a single level of submodules or 'recursive' to get submodules of submodules
  path: string  # path to check out source code, relative to the agent's build directory (e.g. \_work\1)
  persistCredentials: boolean  # set to 'true' to leave the OAuth token in the Git config after the initial fetch

You can also configure fetch depth by setting the Shallow depth option in the pipeline settings UI.

  1. Edit your YAML pipeline and choose More actions, Triggers.

  2. Choose YAML, Get sources.

  3. Configure the Shallow fetch setting. Uncheck Shallow fetch to disable shallow fetch, or check the box and enter a Depth to enable shallow fetch.

Note

If you explicitly set fetchDepth in your checkout step, that setting takes priority over the setting configured in the pipeline settings UI. Setting fetchDepth: 0 fetches all history and overrides the Shallow fetch setting.

You can configure the Shallow fetch setting from the properties of the Get sources task in your pipeline.

In these cases this option can help you conserve network and storage resources. It might also save time. The reason it doesn't always save time is because in some situations the server might need to spend time calculating the commits to download for the depth you specify.

Note

When the pipeline is started, the branch to build is resolved to a commit ID. Then, the agent fetches the branch and checks out the desired commit. There is a small window between when a branch is resolved to a commit ID and when the agent performs the checkout. If the branch updates rapidly and you set a very small value for shallow fetch, the commit may not exist when the agent attempts to check it out. If that happens, increase the shallow fetch depth setting.

Don't sync sources

You may want to skip fetching new commits. This option can be useful in cases when you want to:

You can configure the Don't sync sources setting in the Checkout step of your pipeline, by setting checkout: none.

steps:
- checkout: none  # Don't sync sources

Select the Don't sync sources setting from the properties of the Get sources task in your pipeline.

Note

When you use this option, the agent also skips running Git commands that clean the repo.

Clean build

You can perform different forms of cleaning the working directory of your self-hosted agent before a build runs.

In general, for faster performance of your self-hosted agents, don't clean the repo. In this case, to get the best performance, make sure you're also building incrementally by disabling any Clean option of the task or tool you're using to build.

If you do need to clean the repo (for example to avoid problems caused by residual files from a previous build), your options are below.

Note

Cleaning isn't effective if you're using a Microsoft-hosted agent because you'll get a new agent every time.

You can configure the clean setting in the Checkout step of your pipeline.

steps:
- checkout: self  # self represents the repo where the initial Pipelines YAML file was found
  clean: boolean  # whether to fetch clean each time
  fetchDepth: number  # the depth of commits to ask Git to fetch
  lfs: boolean  # whether to download Git-LFS files
  submodules: true | recursive  # set to 'true' for a single level of submodules or 'recursive' to get submodules of submodules
  path: string  # path to check out source code, relative to the agent's build directory (e.g. \_work\1)
  persistCredentials: boolean  # set to 'true' to leave the OAuth token in the Git config after the initial fetch

When clean is set to true the build pipeline performs an undo of any changes in $(Build.SourcesDirectory). More specifically, the following Git commands are executed prior to fetching the source.

git clean -ffdx
git reset --hard HEAD

For more options, you can configure the workspace setting of a Job.

jobs:
- job: string  # name of the job, A-Z, a-z, 0-9, and underscore
  ...
  workspace:
    clean: outputs | resources | all # what to clean up before the job runs

This gives the following clean options.

Select the Clean setting from the properties of the Get sources task in your pipeline and select one of the following options.

Label sources

You may want to label your source code files to enable your team to easily identify which version of each file is included in the completed build. You also have the option to specify whether the source code should be labeled for all builds or only for successful builds.

You can't currently configure this setting in YAML but you can in the classic editor. When editing a YAML pipeline, you can access the classic editor by choosing either Triggers from the YAML editor menu.

From the classic editor, choose YAML, choose the Get sources task, and then configure the desired properties there.

You can configure the Tag sources setting from the properties of the Get sources task in your pipeline.

In the Tag format you can use user-defined and predefined variables that have a scope of "All." For example:

$(Build.DefinitionName)_$(Build.DefinitionVersion)_$(Build.BuildId)_$(Build.BuildNumber)_$(My.Variable)

The first four variables are predefined. My.Variable can be defined by you on the variables tab.

The pipeline labels your sources with a Git tag.

Some build variables might yield a value that isn't a valid label. For example, variables such as $(Build.RequestedFor) and $(Build.DefinitionName) can contain white space. If the value contains white space, the tag isn't created.

After the sources are tagged by your build pipeline, an artifact with the Git ref refs/tags/{tag} is automatically added to the completed build. This gives your team additional traceability and a more user-friendly way to navigate from the build to the code that was built. The tag is considered a build artifact since it is produced by the build. When the build is deleted either manually or through a retention policy, the tag is also deleted.

Pre-defined variables

When you build a GitHub repository, most of the predefined variables are available to your jobs. However, since Azure Pipelines doesn’t recognize the identity of a user making an update in GitHub, the following variables are set to system identity instead of user's identity:

Status updates

There are two types of statuses that Azure Pipelines posts back to GitHub - basic statuses and GitHub Check Runs. GitHub Checks functionality is only available with GitHub Apps.

Pipeline statuses show up in various places in the GitHub UI.

PAT or OAuth GitHub connections

For pipelines using PAT or OAuth GitHub connections, statuses are posted back to the commit/PR that triggered the run. The GitHub status API is used to post such updates. These statuses contain limited information: pipeline status (failed, success), URL to link back to the build pipeline, and a brief description of the status.

Statuses for PAT or OAuth GitHub connections are only sent at the run level. In other words, you can have a single status updated for an entire run. If you have multiple jobs in a run, you can’t post a separate status for each job. However, multiple pipelines can post separate statuses to the same commit.

GitHub Checks

For pipelines set up using the Azure Pipelines GitHub app, the status is posted back in the form of GitHub Checks. GitHub Checks allow for sending detailed information about the pipeline status and test, code coverage, and errors. The GitHub Checks API can be found here.

For every pipeline using the GitHub App, Checks are posted back for the overall run and each job in that run.

GitHub allows three options when one or more Check Runs fail for a PR/commit. You can choose to "rerun" the individual Check, rerun all the failing Checks on that PR/commit, or rerun all the Checks, whether they succeeded initially or not.

Clicking on the "Rerun" link next to the Check Run name will result in Azure Pipelines retrying the run that generated the Check Run. The resultant run will have the same run number and will use the same version of the source code, configuration, and YAML file as the initial build. Only those jobs that failed in the initial run and any dependent downstream jobs will be run again. Clicking on the "Rerun all failing checks" link will have the same effect. This is the same behavior as clicking "Retry run" in the Azure Pipelines UI. Clicking on "Rerun all checks" will result in a new run, with a new run number and will pick up changes in the configuration or YAML file.

Limitations FAQ

Problems related to GitHub integration fall into the following categories:

Connection types To troubleshoot triggers, how do I know the type of GitHub connection I'm using for my pipeline?

Troubleshooting problems with triggers very much depends on the type of GitHub connection you use in your pipeline. There are two ways to determine the type of connection - from GitHub and from Azure Pipelines.

How do I switch my pipeline to use GitHub app instead of OAuth?

Using a GitHub app instead of OAuth or PAT connection is the recommended integration between GitHub and Azure Pipelines. To switch to GitHub app, follow these steps:

  1. Navigate here and install the app in the GitHub organization of your repository.
  2. During installation, you'll be redirected to Azure DevOps to choose an Azure DevOps organization and project. Choose the organization and project that contain the classic build pipeline you want to use the app for. This choice associates the GitHub App installation with your Azure DevOps organization. If you choose incorrectly, you can visit this page to uninstall the GitHub app from your GitHub org and start over.
  3. In the next page that appears, you don’t need to proceed creating a new pipeline.
  4. Edit your pipeline by visiting the Pipelines page (e.g., https://dev.azure.com/YOUR_ORG_NAME/YOUR_PROJECT_NAME/_build), selecting your pipeline, and clicking Edit.
  5. If this is a YAML pipeline, select the Triggers menu to open the classic editor.
  6. Select the "Get sources" step in the pipeline.
  7. On the green bar with text "Authorized using connection", select "Change" and select the GitHub App connection with the same name as the GitHub organization in which you installed the app.
  8. On the toolbar, select "Save and queue" and then "Save and queue". Select the link to the pipeline run that was queued to make sure it succeeds.
  9. Create (or close and reopen) a pull request in your GitHub repository to verify that a build is successfully queued in its "Checks" section.
Why isn't a GitHub repository displayed for me to choose in Azure Pipelines?

Depending on the authentication type and ownership of the repository, specific permissions are required.

When I select a repository during pipeline creation, I get an error "The repository {repo-name} is in use with the Azure Pipelines GitHub App in another Azure DevOps organization."

This means that your repository is already associated with a pipeline in a different organization. CI and PR events from this repository won't work as they’ll be delivered to the other organization. Here are the steps you should take to remove the mapping to the other organization before proceeding to create a pipeline.

  1. Open a pull request in your GitHub repository, and make the comment /azp where. This reports back the Azure DevOps organization that the repository is mapped to.

  2. To change the mapping, uninstall the app from the GitHub organization, and reinstall it. As you reinstall it, make sure to select the correct organization when you’re redirected to Azure DevOps.

Failing triggers I just created a new YAML pipeline with CI/PR triggers, but the pipeline isn't being triggered.

Follow each of these steps to troubleshoot your failing triggers:

My CI or PR triggers have been working fine. But, they stopped working now.

First, go through the troubleshooting steps in the previous question, then follow these additional steps:

I do not want users to override the list of branches for triggers when they update the YAML file. How can I do this?

Users with permissions to contribute code can update the YAML file and include/exclude additional branches. As a result, users can include their own feature or user branch in their YAML file and push that update to a feature or user branch. This may cause the pipeline to be triggered for all updates to that branch. If you want to prevent this behavior, then you can:

  1. Edit the pipeline in the Azure Pipelines UI.
  2. Navigate to the Triggers menu.
  3. Select Override the YAML continuous Integration trigger from here.
  4. Specify the branches to include or exclude for the trigger.

When you follow these steps, any CI triggers specified in the YAML file are ignored.

Failing checkout I see the following error in the log file during checkout step. How do I fix it? 
remote: Repository not found.
fatal: repository <repo> not found

This could be caused by an outage of GitHub. Try to access the repository in GitHub and make sure that you’re able to.

Wrong version A wrong version of the YAML file is being used in the pipeline. Why is that? Missing status updates My PR in GitHub is blocked since Azure Pipelines didn’t update the status.

This could be a transient error that resulted in Azure DevOps not being able to communicate with GitHub. Retry the check-in GitHub if you use the GitHub app. Or, make a trivial update to the PR to see if the problem can be resolved.

Related articles

RetroSearch is an open source project built by @garambo | Open a GitHub Issue

Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo

HTML: 3.2 | Encoding: UTF-8 | Version: 0.7.4