Depending on your use case, you can set up Continuous Benchmarking in GitHub Actions for your:
Make sure you have created an API token and set it as a Repository secret named BENCHER_API_TOKEN before continuing on! Navigate to Your Repo -> Settings -> Secrets and variables -> Actions -> New repository secret. Name the secret BENCHER_API_TOKEN and set the secret value to your API token.
In GitHub Actions, secrets are not passed to the runner when a workflow is triggered from a forked repository. Therefore, you will need to use a branch from the same repository when adding any of the workflows below to your repository with a pull request. If you try to add Bencher with a pull request from a fork, then the BENCHER_API_TOKEN secret will not be available. ${{ secrets.BENCHER_API_TOKEN }} will be an empty string.
A cornerstone of Statistical Continuous Benchmarking is having a historical baseline for your base branch. This historical baseline can then be used to detect performance regressions in Pull Requests.
on:
push:
branches: main
jobs:
benchmark_base_branch:
name: Continuous Benchmarking with Bencher
permissions:
checks: write
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: bencherdev/bencher@main
- name: Track base branch benchmarks with Bencher
run: |
bencher run \
--project project-abc4567-wxyz123456789 \
--token '${{ secrets.BENCHER_API_TOKEN }}' \
--branch main \
--testbed ubuntu-latest \
--threshold-measure latency \
--threshold-test t_test \
--threshold-max-sample-size 64 \
--threshold-upper-boundary 0.99 \
--thresholds-reset \
--err \
--adapter json \
--github-actions '${{ secrets.GITHUB_TOKEN }}' \
bencher mock
workflow file. (ex: .github/workflows/base_benchmarks.yml)push events to the main branch. See the GitHub Actions on documentation and GitHub Actions push documentation for a full overview. (ex: on: push: branches: main)job. (ex: jobs: benchmark_base_branch)GITHUB_TOKEN to write for checks. (ex: permissions: checks: write)runs-on documentation for a full overview. (ex: runs-on: ubuntu-latest)uses: actions/checkout@v4)uses: bencherdev/bencher@main)bencher run CLI subcommand to run your main branch benchmarks. See the bencher run CLI subcommand for a full overview. (ex: bencher run)--project option to the Project slug. See the --project docs for more details. (ex: --project project-abc4567-wxyz123456789)--token option to the BENCHER_API_TOKEN Repository secret. See the --token docs for more details. (ex: --token '${{ secrets.BENCHER_API_TOKEN }}')--branch option to the base Branch name. See the --branch docs for a full overview. (ex: --branch main)--testbed option to the Testbed name. This should likely match the machine selected in runs-on. See the --tested docs for more details. (ex: --testbed ubuntu-latest)main Branch, ubuntu-latest Testbed, and latency Measure:
--threshold-measure option to the built-in latency Measure that is generated by bencher mock. See the --threshold-measure docs for more details. (ex: --threshold-measure latency)--threshold-test option to a Student’s t-test (t_test). See the --threshold-test docs for a full overview. (ex: --threshold-test t_test)--threshold-max-sample-size option to the maximum sample size of 64. See the --threshold-max-sample-size docs for more details. (ex: --threshold-max-sample-size 64)--threshold-upper-boundary option to the Upper Boundary of 0.99. See the --threshold-upper-boundary docs for more details. (ex: --threshold-upper-boundary 0.99)--thresholds-reset flag so that only the specified Threshold is active. See the --thresholds-reset docs for a full overview. (ex: --thresholds-reset)--err flag to fail the command if an Alert is generated. See the --err docs for a full overview. (ex: --err)--adapter option to Bencher Metric Format JSON (json) that is generated by bencher mock. See benchmark harness adapters for a full overview. (ex: --adapter json)--github-actions option to the GitHub API authentication token to post results as a GitHub Checks comment using the GitHub Actions GITHUB_TOKEN environment variable. See the --github-actions docs for more details. (ex: --github-actions '${{ secrets.GITHUB_TOKEN }}')bencher mock)In order to catch performance regression in Pull Requests, you will need to run your benchmarks on PRs. If you only expect to have PRs from branches within the same repository then you can simply create another workflow to run on pull_request events from the same repository.
⚠️ This solution only works if all PRs are from the same repository! See Pull Requests from Forks below.
on:
pull_request:
types: [opened, reopened, edited, synchronize]
jobs:
benchmark_pr_branch:
name: Continuous Benchmarking PRs with Bencher
# DO NOT REMOVE: For handling Fork PRs see Pull Requests from Forks
if: github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository
permissions:
pull-requests: write
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: bencherdev/bencher@main
- name: Track PR Benchmarks with Bencher
run: |
bencher run \
--project project-abc4567-wxyz123456789 \
--token '${{ secrets.BENCHER_API_TOKEN }}' \
--branch "$GITHUB_HEAD_REF" \
--start-point "$GITHUB_BASE_REF" \
--start-point-hash '${{ github.event.pull_request.base.sha }}' \
--start-point-clone-thresholds \
--start-point-reset \
--testbed ubuntu-latest \
--err \
--adapter json \
--github-actions '${{ secrets.GITHUB_TOKEN }}' \
bencher mock
Create a GitHub Actions workflow file. (ex: .github/workflows/pr_benchmarks.yml)
Run on pull_request events:
opened - A pull request was created.reopened - A previously closed pull request was reopened.edited - The title or body of a pull request was edited, or the base branch of a pull request was changed.synchronize - A pull request’s head branch was updated. For example, the head branch was updated from the base branch or new commits were pushed to the head branch.See the GitHub Actions on documentation and GitHub Actions pull_request documentation for a full overview. (ex: on: pull_request: types: [opened, reopened, edited, synchronize])
Create a GitHub Actions job. (ex: jobs: benchmark_pr_branch)
Run on pull_request events if and only if the pull request is from the same repository. ⚠️ DO NOT REMOVE THIS LINE! For handling Fork PRs see Pull Requests from Forks below. (ex: if: github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository)
Set the permissions for the GITHUB_TOKEN to write for pull-requests. Depending on your GitHub settings, this may not be required. But for all organizations and personal repos created after 02 Feb 2023, this is the default behavior. See the GitHub documentation for a full overview. (ex: permissions: pull-requests: write)
Set the type of machine the job will run on. See the GitHub Actions runs-on documentation for a full overview. (ex: runs-on: ubuntu-latest)
Checkout the PR branch source code. (ex: uses: actions/checkout@v4)
Install the Bencher CLI using the GitHub Action. (ex: uses: bencherdev/bencher@main)
Use the bencher run CLI subcommand to run your pull request branch benchmarks. See the bencher run CLI subcommand for a full overview. (ex: bencher run)
Set the --project option to the Project slug. See the --project docs for more details. (ex: --project project-abc4567-wxyz123456789)
Set the --token option to the BENCHER_API_TOKEN Repository secret. See the --token docs for more details. (ex: --token '${{ secrets.BENCHER_API_TOKEN }}')
Set the --branch option to the PR branch name using the GitHub Actions GITHUB_HEAD_REF default environment variable. See the --branch docs for a full overview. (ex: --branch "$GITHUB_HEAD_REF")
Set the Start Point for the PR Branch:
--start-point option to the PR Branch start point using the GitHub Actions GITHUB_BASE_REF default environment variable. See the --start-point docs for a full overview. (ex: --start-point "$GITHUB_BASE_REF")--start-point-hash option to the PR Branch start point git hash using the GitHub Actions pull_request event. See the --start-point-hash docs for a full overview. (ex: --start-point-hash '${{ github.event.pull_request.base.sha }}')--start-point-clone-thresholds flag to clone the Thresholds from the start point. See the --start-point-clone-thresholds docs for a full overview. (ex: --start-point-clone-thresholds)--start-point-reset flag to always reset the PR Branch to the start point. This will prevent benchmark data drift. See the --start-point-reset docs for a full overview. (ex: --start-point-reset)Set the --testbed option to the Testbed name. This should likely match the machine selected in runs-on. See the --tested docs for more details. (ex: --testbed ubuntu-latest)
Set the --err flag to fail the command if an Alert is generated. See the --err docs for a full overview. (ex: --err)
Set the --adapter option to Bencher Metric Format JSON (json) that is generated by bencher mock. See benchmark harness adapters for a full overview. (ex: --adapter json)
Set the --github-actions option to the GitHub API authentication token to post results as a comment on the Pull Request using the GitHub Actions GITHUB_TOKEN environment variable. See the --github-actions docs for more details. (ex: --github-actions '${{ secrets.GITHUB_TOKEN }}')
Specify the benchmark command arguments. See benchmark command for a full overview. (ex: bencher mock)
To clean up the PR branch after its PR is closed, you can create a separate workflow to run on pull_request events with the closed type. This workflow will archive the PR branch using the bencher archive command.
on:
pull_request:
types: [closed]
jobs:
archive_pr_branch:
name: Archive closed PR branch with Bencher
# DO NOT REMOVE: For handling Fork PRs see Pull Requests from Forks
if: github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: bencherdev/bencher@main
- name: Archive closed PR branch with Bencher
run: |
bencher archive \
--project project-abc4567-wxyz123456789 \
--token '${{ secrets.BENCHER_API_TOKEN }}' \
--branch "$GITHUB_HEAD_REF"
Create a GitHub Actions workflow file. (ex: .github/workflows/pr_benchmarks_closed.yml)
Run on pull_request events:
closed - A pull request was closed.See the GitHub Actions on documentation and GitHub Actions pull_request documentation for a full overview. (ex: on: pull_request: types: [closed])
Create a GitHub Actions job. (ex: jobs: archive_pr_branch)
Run on pull_request events if and only if the pull request is from the same repository. ⚠️ DO NOT REMOVE THIS LINE! For handling Fork PRs see Pull Requests from Forks below. (ex: if: github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository)
Set the type of machine the job will run on. See the GitHub Actions runs-on documentation for a full overview. (ex: runs-on: ubuntu-latest)
Checkout the PR branch source code. (ex: uses: actions/checkout@v4)
Install the Bencher CLI using the GitHub Action. (ex: uses: bencherdev/bencher@main)
Use the bencher archive CLI subcommand to archive the PR branch. (ex: bencher archive)
Set the --project option to the Project slug. See the --project docs for more details. (ex: --project project-abc4567-wxyz123456789)
Set the --token option to the BENCHER_API_TOKEN Repository secret. See the --token docs for more details. (ex: --token '${{ secrets.BENCHER_API_TOKEN }}')
Set the --branch option to the PR branch name using the GitHub Actions GITHUB_HEAD_REF default environment variable. (ex: --branch "$GITHUB_HEAD_REF")
If you plan to accept pull requests from forks, as is often the case in public open source projects, then you will need to handle things a little differently. For security reasons, secrets such as your BENCHER_API_TOKEN and the GITHUB_TOKEN are not available in GitHub Actions for fork PRs. That is if an external contributor opens up a PR from a fork the above example will not work. See this GitHub Security Lab write up and this blog post on preventing pwn requests for a full overview.
This is the safe and suggested way to add Continuous Benchmarking to fork pull requests. It requires two separate workflows. The first workflow runs and caches the benchmark results in the pull_request context. No secrets such as your BENCHER_API_TOKEN and the GITHUB_TOKEN are available there. Then a second workflow downloads the cached benchmark results in the workflow_run context and uploads them to Bencher. This works because workflow_run runs in the context of the repository’s default branch, where secrets such as your BENCHER_API_TOKEN and the GITHUB_TOKEN are available. The pull request number, head branch, and base branch used in the initial pull_request workflow must also be explicitly passed into the workflow_run workflow since they are not available there. These workflows will only run if they exist on the default branch. See using data from the triggering workflow for a full overview.
name: Run Benchmarks
on:
pull_request:
types: [opened, reopened, edited, synchronize]
jobs:
benchmark_fork_pr_branch:
name: Run Fork PR Benchmarks
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Mock Benchmarking
run: |
/bin/echo '{ "bencher::mock_0": { "latency": { "value": 1.0 } } }' > benchmark_results.json
- name: Upload Benchmark Results
uses: actions/upload-artifact@v4
with:
name: benchmark_results.json
path: ./benchmark_results.json
- name: Upload GitHub Pull Request Event
uses: actions/upload-artifact@v4
with:
name: event.json
path: ${{ github.event_path }}
Create a first GitHub Actions workflow file. (ex: .github/workflows/fork_pr_benchmarks_run.yml)
Name this workflow so it can be referenced by the second workflow. (ex: name: Run Benchmarks)
Run on pull_request events:
opened - A pull request was created.reopened - A previously closed pull request was reopened.edited - The title or body of a pull request was edited, or the base branch of a pull request was changed.synchronize - A pull request’s head branch was updated. For example, the head branch was updated from the base branch or new commits were pushed to the head branch.See the GitHub Actions on documentation and GitHub Actions pull_request documentation for a full overview. (ex: on: pull_request: types: [opened, reopened, edited, synchronize])
Create a GitHub Actions job. (ex: jobs: benchmark_fork_pr_branch)
Set the type of machine the job will run on. See the GitHub Actions runs-on documentation for a full overview. (ex: runs-on: ubuntu-latest)
Checkout the fork PR branch source code. (ex: uses: actions/checkout@v4)
Run your benchmarks and save the results to a file. (ex: /bin/echo '{ ... }' > benchmark_results.json)
Upload the benchmark results file as an artifact. (ex: uses: actions/upload-artifact@v4)
Upload the pull_request event object as an artifact. (ex: uses: actions/upload-artifact@v4)
name: Track Benchmarks with Bencher
on:
workflow_run:
workflows: [Run Benchmarks]
types: [completed]
jobs:
track_fork_pr_branch:
if: github.event.workflow_run.conclusion == 'success'
permissions:
pull-requests: write
runs-on: ubuntu-latest
env:
BENCHMARK_RESULTS: benchmark_results.json
PR_EVENT: event.json
steps:
- name: Download Benchmark Results
uses: dawidd6/action-download-artifact@v6
with:
name: ${{ env.BENCHMARK_RESULTS }}
run_id: ${{ github.event.workflow_run.id }}
- name: Download PR Event
uses: dawidd6/action-download-artifact@v6
with:
name: ${{ env.PR_EVENT }}
run_id: ${{ github.event.workflow_run.id }}
- name: Export PR Event Data
uses: actions/github-script@v6
with:
script: |
let fs = require('fs');
let prEvent = JSON.parse(fs.readFileSync(process.env.PR_EVENT, {encoding: 'utf8'}));
core.exportVariable("PR_HEAD", prEvent.pull_request.head.ref);
core.exportVariable("PR_HEAD_SHA", prEvent.pull_request.head.sha);
core.exportVariable("PR_BASE", prEvent.pull_request.base.ref);
core.exportVariable("PR_BASE_SHA", prEvent.pull_request.base.sha);
core.exportVariable("PR_NUMBER", prEvent.number);
- uses: bencherdev/bencher@main
- name: Track Benchmarks with Bencher
run: |
bencher run \
--project project-abc4567-wxyz123456789 \
--token '${{ secrets.BENCHER_API_TOKEN }}' \
--branch "$PR_HEAD" \
--hash "$PR_HEAD_SHA" \
--start-point "$PR_BASE" \
--start-point-hash "$PR_BASE_SHA" \
--start-point-clone-thresholds \
--start-point-reset \
--testbed ubuntu-latest \
--err \
--adapter json \
--github-actions '${{ secrets.GITHUB_TOKEN }}' \
--ci-number "$PR_NUMBER" \
--file "$BENCHMARK_RESULTS"
workflow file. (ex: .github/workflows/fork_pr_benchmarks_track.yml)name: Track Benchmarks with Bencher)workflow_run event. (ex: on: workflow_run: ...)job. (ex: jobs: track_fork_pr_branch)workflow_run event. (ex: if: github.event.workflow_run.conclusion == 'success')GITHUB_TOKEN to write for pull-requests. Depending on your GitHub settings, this may not be required. But for all organizations and personal repos created after 02 Feb 2023, this is the default behavior. See the GitHub documentation for a full overview. (ex: permissions: pull-requests: write)runs-on documentation for a full overview. (ex: runs-on: ubuntu-latest)pull_request event object file names as environment variables. (ex: env: ...)pull_request event using the action-download-artifact GitHub Action. (ex: uses: dawidd6/action-download-artifact@v6)pull_request event as intermediate environment variables. (ex: core.exportVariable(...))uses: bencherdev/bencher@main)bencher run CLI subcommand to track your fork pull branch benchmarks. See the bencher run CLI subcommand for a full overview. (ex: bencher run)--project option to the Project slug. See the --project docs for more details. (ex: --project project-abc4567-wxyz123456789)--token option to the BENCHER_API_TOKEN Repository secret. See the --token docs for more details. (ex: --token '${{ secrets.BENCHER_API_TOKEN }}')--branch option to the fork PR branch name using an intermediate environment variable. See the --branch docs for a full overview. (ex: --branch "$PR_HEAD")--hash option to the fork PR branch git hash using an intermediate environment variable. See the --hash docs for a full overview. (ex: --hash "$PR_HEAD_SHA")--start-point option to the fork PR Branch start point using an intermediate environment variable. See the --start-point docs for a full overview. (ex: --start-point "$PR_BASE")--start-point-hash option to the fork PR Branch start point git hash using an intermediate environment variable. See the --start-point-hash docs for a full overview. (ex: --start-point-hash "$PR_BASE_SHA")--start-point-clone-thresholds flag to clone the Thresholds from the start point. See the --start-point-clone-thresholds docs for a full overview. (ex: --start-point-clone-thresholds)--start-point-reset flag to always reset the fork PR Branch to the start point. This will prevent benchmark data drift. See the --start-point-reset docs for a full overview. (ex: --start-point-reset)--testbed option to the Testbed name. This should likely match the machine selected in runs-on. See the --tested docs for more details. (ex: --testbed ubuntu-latest)--err flag to fail the command if an Alert is generated. See the --err docs for a full overview. (ex: --err)--adapter option to Bencher Metric Format JSON (json) that is generated by bencher mock. See benchmark harness adapters for a full overview. (ex: --adapter json)--github-actions option to the GitHub API authentication token to post results as a comment on the Pull Request using the GitHub Actions GITHUB_TOKEN environment variable. See the --github-actions docs for more details. (ex: --github-actions '${{ secrets.GITHUB_TOKEN }}')--ci-number option to the pull request number using an intermediate environment variable. See the --ci-number docs for more details. (ex: --ci-number "$PR_NUMBER")--file option to the benchmark results file path. See benchmark command for a full overview. (ex: --file "$BENCHMARK_RESULTS")To clean up the fork PR branch after its PR is closed, you can create a separate workflow to run on pull_request_target events with the closed type. This workflow will archive the fork PR branch using the bencher archive command.
on:
pull_request_target:
types: [closed]
jobs:
archive_fork_pr_branch:
name: Archive closed fork PR branch with Bencher
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: bencherdev/bencher@main
- name: Archive closed fork PR branch with Bencher
run: |
bencher archive \
--project project-abc4567-wxyz123456789 \
--token '${{ secrets.BENCHER_API_TOKEN }}' \
--branch "$GITHUB_HEAD_REF"
Create a GitHub Actions workflow file. (ex: .github/workflows/fork_pr_benchmarks_closed.yml)
Run on pull_request_target events:
closed - A pull request was closed.See the GitHub Actions on documentation and GitHub Actions pull_request_target documentation for a full overview. (ex: on: pull_request_target: types: [closed])
Create a GitHub Actions job. (ex: jobs: archive_pr_branch)
Set the type of machine the job will run on. See the GitHub Actions runs-on documentation for a full overview. (ex: runs-on: ubuntu-latest)
Checkout the PR branch source code. (ex: uses: actions/checkout@v4)
Install the Bencher CLI using the GitHub Action. (ex: uses: bencherdev/bencher@main)
Use the bencher archive CLI subcommand to archive the PR branch. (ex: bencher archive)
Set the --project option to the Project slug. See the --project docs for more details. (ex: --project project-abc4567-wxyz123456789)
Set the --token option to the BENCHER_API_TOKEN Repository secret. See the --token docs for more details. (ex: --token '${{ secrets.BENCHER_API_TOKEN }}')
Set the --branch option to the PR branch name using the GitHub Actions GITHUB_HEAD_REF default environment variable. (ex: --branch "$GITHUB_HEAD_REF")
🐰 Congrats! You have learned how to use Bencher in GitHub Actions! 🎉
Published: Sat, August 12, 2023 at 4:07:00 PM UTC | Last Updated: Sat, August 2, 2025 at 7:40:00 AM UTC
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4