Sourcegraph ships with Grafana for dashboards, Prometheus for metrics and alerting. We also provide built-in alerting for these metrics.
GrafanaSite admins can view the Grafana monitoring dashboards on a Sourcegraph instance:
https://sourcegraph.example.com/-/debug/grafana/
.A complete dashboard reference is available for more context on our available service dashboards and panels.
Additional dashboards can also be set up—see Grafana configuration for more details.
View documentationOn service dashboards, each metric panel has links attached that lead to relevant alert solutions or panel documentation. These can be accessed from the top left corner of each panel.
View alertsThe Overview dashboard includes a high-level summary of alert events across all Sourcegraph services.
On service dashboards, a summary of alert events for that service is available. Alerts can be filtered by level using the "Filter alert level" selector at the top of the dashboard. This information can be overlaid on all panels in that dashboard by enabling the "Alert events" toggle.
View version change eventsOn service dashboards, annotations can be displayed for occasions when a Sourcegraph version change is detected by enabling the "Version changes" toggle at the top of the dashboard. This can be useful for correlating metrics and alerts with Sourcegraph deployment and upgrade events.
Querying metricsSpecific metrics can be queried using Grafana's Explore panel, available at /-/debug/grafana/explore
on your Sourcegraph instance. The query for each individual panel can be viewed and explored by clicking on the panel and using the "Explore" option.
Sourcegraph deploys a customized image of Grafana, which ships with Sourcegraph-specific dashboard definitions.
To provide custom dashboards, a directory containing dashboard JSON specifications can be mounted in the Docker container at /sg_grafana_additional_dashboards
. Changes to files in that directory will be detected automatically while Grafana is running.
More behavior can be controlled with environmental variables.
Accessing Grafana directlyAs a workaround, site admins can connect to Grafana directly to make changes using the Grafana UI.
For most use cases, you can access Grafana through your Sourcegraph instance. Follow the instructions below to access Grafana directly to, for example, edit configuration directly.
NOTE: Most of the dashboards that Sourcegraph ships with are not configurable through the Grafana UI. In general, we recommend these configuration methods instead.
If you are using the Kubernetes deployment option, you can access Grafana directly using Kubernetes port forwarding to your local machine:
kubectl port-forward svc/grafana 3370:30070
Grafana will be available http://localhost:3370/-/debug/grafana.
If you are using Docker Single Container or the Docker Compose deployment option, Grafana is available locally at http://localhost:3370/-/debug/grafana without any additional setup. If Sourcegraph is deployed to a remote server, then access via an SSH tunnel using a tool such as sshuttle is required to establish a secure connection to Grafana. To access the remote server using sshuttle
from your local machine:
sshuttle -r user@host 0/0
Grafana will be available at http://host:3370/-/debug/grafana.
Grafana securityTo enable Grafana's built-in authentication, add GF_AUTH_ANONYMOUS_ENABLED
as false
in the Grafana container's environment variables in your deployment override file, then redeploy the Grafana container.
We also recommend that you customize the default admin username and password.
For new deployments, configure the GF_SECURITY_ADMIN_USER
and GF_SECURITY_ADMIN_PASSWORD
environment variables on the Grafana container, using your deployment pipeline's secrets management tool, before initial deployment, to initialize the default admin user on first deployment.
For existing deployments, you can either log in to the Grafana web UI with the default credentials and it will prompt you to change the password, or you can reinitialize Grafana:
The contents of the Grafana container's storage volume are ephemeral, so metrics data is not lost in this process.
# Helm override
grafana:
env:
GF_AUTH_ANONYMOUS_ENABLED:
value: "false"
GF_SECURITY_ADMIN_USER:
value: "custom-admin-username"
GF_SECURITY_ADMIN_PASSWORD:
value: "custom-admin-password"
# Docker Compose override
grafana:
environment:
- 'GF_AUTH_ANONYMOUS_ENABLED=false'
- 'GF_SECURITY_ADMIN_USER=custom-admin-username'
- 'GF_SECURITY_ADMIN_PASSWORD=custom-admin-password'
Prometheus
Prometheus is a monitoring tool that collects application- and system-level metrics over time and makes these accessible through a robust query language.
For most use cases, you can query Prometheus through Grafana using Grafana's Explore panel, available at /-/debug/grafana/explore
on your Sourcegraph instance, or simply rely on the dashboards we ship.
Sourcegraph's metrics include a single high-level metric alert_count
which indicates the number of level=critical
and level=warning
alerts each service has fired over time for each Sourcegraph service. This is the same metric presented on the Overview Grafana dashboard.
NOTE: We provide built-in alerting for these alerting metrics to help monitor the health of your Sourcegraph instance. Refer to our alert solutions reference for details on specific alerts.
Description: The number of alerts each service has fired for a given alert name and severity level.
Values: Although the values of alert_count
are floating-point numbers, only their whole numbers have meaning. For example, 0.5
and 0.7
indicate no alerts are firing, while 1.2
indicates exactly one alert is firing and 3.0
indicates exactly three alerts firing.
Labels:
Label Descriptionservice_name
the name of the service that fired the alert name
the name of the alert that the service fired level
either critical
or warning
, as defined here description
a human-readable description of the alert Complete reference
A complete reference of Sourcegraph's vast set of Prometheus metrics is not yet available. If you are interested in this, please reach out by filing an issue or contacting us at [email protected].
Prometheus configurationSourcegraph runs a customized image of Prometheus, which packages a standard Prometheus installation together with rules files and target files tailored to Sourcegraph and quality-of-life integrations such as the ability to configure alerting from the Sourcegraph web application.
A directory can be mounted at /sg_prometheus_add_ons
. It can contain additional config files of two types:
_rules.yml
in their filename (ie gitserver_rules.yml
)_targets.yml
in their filename (ie local_targets.yml
)Rule files and target files must use the latest Prometheus 2.x syntax.
The environment variable PROMETHEUS_ADDITIONAL_FLAGS
can be used to pass on additional flags to the prometheus
executable running in the container.
Most of the time, Sourcegraph site admins will monitor and query key metrics through Grafana, rather than through Prometheus directly. Grafana also provides the dashboards that monitor the standard metrics that indicate the health of the instance. Follow the instructions below to access Prometheus directly instead.
If you are using the Kubernetes deployment option, port-forward the Prometheus service:
kubectl port-forward svc/prometheus 9090:30090
If you are using Docker Single Container or the Docker Compose deployment option, you will need to restart the Sourcegraph container with a flag --publish 9090:9090
in the docker run
command.
Prometheus will be available http://localhost:9090.
Using a custom monitoring systemPlease refer to our FAQ item, "Can I consume Sourcegraph's metrics in my own monitoring system (Datadog, New Relic, etc.)".
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4