The toolbox is an open source package of command line tools available in every Semaphore job. It provides utility functions to clone, cache data, run test services, or change language versions. This page explains all the available tools.
OverviewâThe Semaphore toolbox is preinstalled in all Semaphore jobs. It is also installed in self-hosted agents, so you can assume that all the tools described on this page are available for use.
artifactâThis is the command line tool to access the Semaphore artifacts.
The usage is:
artifact <command> <namespace> <file_or_directory> [flags]
The available commands are:
help: print help on the terminalpull: pull a file or directory from the artifact storepush: push a file or directory to the artifact storeyank: pull and delete a file or directory from the artifact storeThe available namespaces are:
job: storage available only to current jobworkflow: storage available to jobs in all pipelines in the workflowproject: global storage for the projectSee artifact namespaces for more details.
The optional flags are:
--force or -f: overwrite file or directory if already exists--destination of -d: pull or yank the file into a different path--verbose or -v: verbose loggingArtifact usage examples
# force push a final deliverable to the project level
artifact push project app-v1.tar.gz --force
# push a screenshot to the job level
artifact push job screenshots/app.png
# push a binary to the workflow level
artifact push workflow build/app
# pull a binary from the workflow level
artifact pull workflow build/app
# delete a binary from the workflow level
artifact yank workflow build/app
The artifact pull and artifact yank commands exit with non-zero status (ending the job with error) when:
-f or --force is supplied)The uploaded files must meet the following requirements:
.well-known/acme-challenge/{, }, |, \, ^, ~, [, ]. or ...You can workaround these limitations by compressing the file with tar before pushing it to the artifact store. For example:
Creating a tarball before storing the artifact
tar -czvf example.tar.gz ~/example
artifact push workflow example.tar.gz
To retrieve the files, use:
Retrieving the tarball
artifact pull workflow example.tar.gz
tar -xzf example.tar.gz
The cache tool lets you interact with your project's Semaphore cache.
The syntax is:
cache <command> <keys> [path] [flags]
The available commands are:
clear: remove all keys in the cache.delete: delete a key from the cache.has_key: check if a key is present in the cache. Exits with non-zero status if the key is not found.is_not_empty: check if the cache is not empty.list: list all keys in the cache.restore: restore keys from the cache.store: store keys in the cache.usage: get a summary of cache usage.The cache keys are a comma-separated list of strings to label or select cached items.
The following examples show different ways to store and identify Ruby Gems in the vendor/bundle directory:
cache store our-gems vendor/bundle
cache store gems-$SEMAPHORE_GIT_BRANCH vendor/bundle
cache store gems-$SEMAPHORE_GIT_BRANCH-revision-$(checksum Gemfile.lock) vendor/bundle
To restore counterparts for the example above are:
cache restore our-gems
cache restore gems-$SEMAPHORE_GIT_BRANCH
cache restore gems-$SEMAPHORE_GIT_BRANCH-revision-$(checksum Gemfile.lock),gems-master
cache restore always retrieves the first item that matches a key found in the cache. The rest are ignored. If no archives are restored the command exits with exit status 0.
note
As the cache store uses tar, which automatically removes any leading / from a given path value, any further changes of the path after the store command completes will not be automatically propagated to the cache.
Space managementâThe cache store command automatically deletes older files when the cache is full. You can change the deletion criteria using the --cleanup-by or -c argument. For example:
cache store our-gems vendor/bundle --cleanup-by SIZE
The supported options for --cleanup-by are:
SIZE: delete biggest files firstSTORE_TIME: (default) delete oldest files firstACCESS_TIME: delete oldest accessed files firstThe cache tool depends on the following environment variables:
checkoutâThis tool clones the repository using Git. For performance reasons, the default behavior is to perform a shallow clone. Shallow clones only include the latest commit instead of the full repository history.
The checkout command takes no arguments. It also changes the directory into the cloned repository. So, for example, this would work on a Node.js project:
No need to cd into the cloned repository
Full cloneâIf you want to do a full clone of the repository you can run the following commands to "unshallow" a repository:
checkout
git fetch --unshallow
git config remote.origin.fetch '+refs/heads/*:refs/remotes/origin/*'
git fetch --all
info
This is marked as an experimental feature.
If a full clone takes too long, you can use checkout --use-cache to save a copy of the repository in the Semaphore cache. This should speed up full clones significantly.
To perform a full repository clone with cache run:
This option creates or refreshes one or more copies of the repository in the Semaphore cache. See the environment variables to learn how to control the behavior of the cached copies.
Environment variablesâThe checkout command uses the following environment variables.
SEMAPHORE_GIT_URLSEMAPHORE_GIT_DIRSEMAPHORE_GIT_SHASEMAPHORE_GIT_DEPTHSEMAPHORE_GIT_CACHE_AGE (only available after using --use-cache)SEMAPHORE_GIT_CACHE_KEEP (only available after using --use-cache)This tool takes a single argument which is the file to checksum. It outputs the MD5 checksum of the file's contents. This tool is useful for tagging artifacts or generating cache keys.
The syntax is:
Example:
Checksumming package-lock.json
$ checksum package-lock.json
3dc6f33834092c93d26b71f9a35e4bb3
The install-package tool is used to manage Ubuntu packages you may need for your jobs. It downloads and caches packages in a way that can be quickly reinstalled over and over again in different jobs. This is a convenient tool, you can still use sudo to install packages using the system's package manager.
The syntax is:
install-package syntax
install-package <command> <pkg-name>
Where <pkg-name> is the Ubuntu package name without the .deb extension. In other words, you want libc6 instead of libc6.deb.
Where command is one of the following:
update: Retrieve new lists of packagesupgrade: Perform an upgradeinstall: Install new packagesreinstall: Reinstall packagesremove: Remove packagespurge: Remove packages and config filesautoremove: Remove automatically all unused packagesdist-upgrade: Distribution upgrade, see apt-get(8)dselect-upgrade: Follow dselect selectionsbuild-dep: Configure build-dependencies for source packagessatisfy: Satisfy dependency stringsclean: Erase downloaded archive filesautoclean: Erase old downloaded archive filescheck: Verify that there are no broken dependenciessource: Download source archivesdownload: Download the binary package into the current directorychangelog: Download and display the changelog for the given packageYou can supply multiple packages with their versions in the same invocation:
install-package install mongodb-clients=3.6.8 mysql-client=8.0.36-0ubuntu0.20.04.1
The tool integrates with the Semaphore cache to save, retrieve, and update the Deb packages as needed.
You can reinstall the packages in a different job within the same project with:
Installing packages from the cache in other jobs
retryâThe retry tool can be used to retry a command on an interval. This is useful when waiting for resources to become available or to mitigate connectivity issues.
The syntax is:
Where flags are optional arguments:
--times or -t: number of times to retry before giving up. Default is 3--sleep or -s: wait interval between retries in seconds. Default is 0For example, to retry 5 times bundle install with a 10-second sleep use:
retry --times 5 --sleep 10 bundle install
This tool enables sharing key-value data between Semaphore jobs within the same workflow. It carries values from parent jobs, and updates made within a workflow are inherited by promoted jobs. However, please note that values are not shared across different promotions.
note
sem-context depends on artifacts to store data.
The syntax is:
sem-context syntax
sem-context <action> <key>[=value] [flags]
The possible actions are:
put: store a key-value pairget: retrieve a value for a keydelete: delete a keyThe following limitations apply to keys and values:
- and _ are allowed) of length 3 to 256 charactersPossible flags are:
--ignore-failure forces the tool to exit with exit status 0 on failure--force overwrites existing keys--fallback <value> provides a default value for the get action if the key is not presentThe following example stores the version number with the key ReleaseVersion.
sem-context put ReleaseVersion=1.2.3
Exit status codes:
The following example retrieves the version number using the key ReleaseVersion.
$ sem-context get ReleaseVersion
1.2.3
Exit status codes:
The following example deletes the version value using the key ReleaseVersion.
sem-context delete ReleaseVersion
Exit status codes:
The sem-service tool manages databases and other useful services in Ubuntu-based environments.
The syntax is:
sem-service syntax
sem-service <command> <service> [version] [flags]
Where <command> is one of the following:
start: starts service in the background, returns when the service is ready to accept connectionsstop: stops a running servicestatus: returns non-zero exit status if service is not runningAll services run locally on the agent running the job. Started services always listen on 0.0.0.0 and their default port.
Services supportedâThe <service> argument is one of the following:
mysqlpostgrespostgisredisrabbitmqmemcachedmongodbelasticsearchopensearchcassandrarethinkdbThe <version> argument depends on the kind of service being started. You can run sem-service help to see all available services and versions.
When starting mysql or postgres services you can provide the following optional arguments:
--username=<username>: user with admin permissions on the instance
mysql defaults to "root"postgres defaults to "postgres"--password=<password>: password for the admin user
mysql defaults to "test"postgres defaults to a blank string--db=<name> database name to create and default toThe sem-service tool pulls images from the Semaphore Container Registry.
The sem-version tool manages language and utility versions in Ubuntu environments. It provides a quick and simple way to install and switch to a version of a command line tool or programming language.
The syntax is:
sem-version syntax
sem-version <target> <version> [flags]
Where <target> is one of the following:
elixirerlanggojavakubectlphprubypythonscalanodeThe <version> depends on the target used. The command fails with a non-zero exit code unless the --ignore or -i flag is supplied.
For example, to download and use Go version 1.22:
spcâThe Semaphore Pipeline Compiler (SPC) is used during initialization jobs to process runtime values in input pipelines. It generates an output pipeline that is used for the workflow execution.
note
spc is called automatically by Semaphore during initialization.
The syntax is:
spc syntax
spc compile --input <input-pipeline-file> --output <output-pipeline-file> --logs <logs-file>
The tool evaluates change_in arguments and outputs the result of each match condition.
This tool processes test results in the JUnit format. It is used to process and generate test reports and the flaky tests dashboard.
The syntax is:
test-results <command> [flags]
The available commands are:
combine: combines multiples JSON report files into one file summarycompile: parses JUnit XML files to a well-defined JSON report filegen-pipeline-report: fetches and combines workflow-level JUnit report and combines thempublish: parses XML JUnit file to a well-defined JSON schema and publishes results to artifacts storagecommand-metrics: generates a Markdown-compatible Mermaid.js Gantt diagram showing the job commands timeline. Used to produce Markdown Reportsresource-metrics: generates a Markdown-compatible computer resource report for the job. Used to produce Markdown ReportsThe available flags are:
--config <path>: path to the config file. Default is $HOME/.test-results.yaml--name <name> or -N <name>: override the name of the test suite--parser <name> or -p <name>: override the parser used. Defaults to "auto"--suite-prefix <name> or -S <name>: add a prefix for the test suite name--trace: provide trace loggingThe test-results CLI is open-sourced and available on semaphoreci/test-results. You can access the source code, contribute, and report issues there.
Merging test resultsâTo use the test result feature you must add the following command at the end of every test job.
The syntax is:
test-results publish <path>
Where <path> is a file or a directory containing JUnit XML files. This processes the files and uploads the results to the artifact store.
If working with Docker containers, you need to expose the JUnit files using bind mounts. For example:
docker run -v $(pwd):/app test-runner run-specs
test-results publish junit.xml
sem-semantic-release wraps the [semantic release CLI] and exports the release information to the rest of your delivery process using sem-context put. This information can later be retrieved with sem-context get.
It is handy when you want to continue the delivery process once the release has been created. A common example would be to build a Docker image tagged with the release version, or add an annotation to a Kubernetes deployment manifest.
General usage in CI:
checkout
sem-semantic-release
Usage options:
Usage: sem-semantic-release [OPTION]...
Options:
--dry-run runs semantic-release without publishing version
--plugins npm plugins and extensions to be installed
--branches branches to run semantic release for
--version semantic-release version
The sem-semantic-release exports the following values to sem-context
ReleasePublished Whether a new release was published ("true" or "false") ReleaseVersion Version of the new release. (e.g. "1.3.0") ReleaseMajorVersion Major version of the new release. (e.g. "1") ReleaseMinorVersion Minor version of the new release. (e.g. "3") ReleasePatchVersion Patch version of the new release. (e.g. "0")
You can use these values in your pipeline to automatically figure version numbers. For example:
docker build . -t my-app:$(sem-context get ReleaseVersion)
docker push
sed -i "s/IMAGE/$(sem-context get ReleaseVersion)" deployment.yml
kubectl apply -f deployment.yml
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4