This document describes how to manage services and service-level objectives (SLOs) by using the Cloud Monitoring API.
Service Monitoring adds the following resources to the Monitoring API:
This document refers to these additions collectively as the SLO API and illustrates the primary uses. For a general overview of the structures in the SLO API, see Constructs in the API. Comprehensive reference material appears under Cloud Monitoring API v3.
You can use the SLO API to do the following:
Create and manage services.
Create service-level objectives (SLOs) based on custom or predefined service-level indicators (SLIs) for any of your services.
Several methods in the SLO API use identifiers for different elements, particularly identifiers for project and services. These IDs adhere to the following rules:
The regular expression for these rules is as follows: [a-z](?:[-a-z0-9]{0,61}[a-z0-9])?
curl
This section describes the conventions and setup used for invoking the SLO API by using the curl
tool. If you are using this API through a client library, you can skip this section.
The examples on this page access the SLO API by using the curl
tool to send HTTP requests to REST endpoints. Use the following information about authentication and about invoking curl
to set the variables used in the invocations.
Create an environment variable to hold the ID of your scoping project of a metrics scope: These examples use PROJECT_ID
:
PROJECT_ID=my-test-service
Authenticate to the Google Cloud CLI:
gcloud auth login
To avoid having to specify your project ID with each command, set it as the default by using gcloud CLI:
gcloud config set project ${PROJECT_ID}
Create an authorization token and capture it in an environment variable. These examples call the variable ACCESS_TOKEN
:
ACCESS_TOKEN=`gcloud auth print-access-token`
You have to periodically refresh the access token. If commands that worked suddenly report that you are unauthenticated, re-issue this command.
To verify that you got an access token, echo the ACCESS_TOKEN
variable:
echo ${ACCESS_TOKEN}
ya29.GluiBj8o....
curl
Each curl
invocation includes a set of arguments, followed by the URL of a SLO API resource. The common arguments include values specified by the PROJECT_ID
and ACCESS_TOKEN
environment variables. You might also have to specify other arguments, for example, to specify the type of the HTTP request (for example, -X DELETE
). The default request is GET
, so the examples don't specify it.
Each curl
invocation has this general structure:
curl --http1.1 --header "Authorization: Bearer ${ACCESS_TOKEN}" <other_args> https://monitoring.googleapis.com/v3/projects/${PROJECT_ID}/<request>
For example, to list all the services in your project, issue the following GET
request:
curl --http1.1 --header "Authorization: Bearer ${ACCESS_TOKEN}" https://monitoring.googleapis.com/v3/projects/${PROJECT_ID}/services
This request returns an array of service descriptions, with entries like the following, a GKE workload service with the service ID “my-test-service”:
{ "services": [ { "name": "projects/PROJECT_NUMBER/services/my-test-service", "displayName": "My Test GKE Workload Service", "basicService": { "serviceType": "GKE_WORKLOAD", "serviceLabels": { "cluster_name": "workload-test", "location": "us-central1-c", "namespace_name": "kube-system", "project_id": "lesser-weevil", "top_level_controller_name": "gke-metrics-controller", "top_level_controller_type": "DaemonSet" } }, "gkeWorkload": { "projectId": "lesser-weevil", "location": "us-central1-c", "clusterName": "workload-test", "namespaceName": "kube-system", "topLevelControllerType": "DaemonSet", "topLevelControllerName": "gke-metrics-controller" }, "source": "USER", "telemetry": { "resourceName": "//container.googleapis.com/projects/PROJECT_ID/zones/us-central1-c/clusters/workload-test/k8s/namespaces/kube-system/apps/daemonsets/gke-metrics-controller" } }, ... ] }
To see the configuration used to create this service, see Creating a service. Note that the gkeWorkload
structure in this listing is derived from the basicService
structure used to create the service. See Service
for more information about the structure of a service.
The Service
resource acts as the root element for organizing your services. Aspects of a particular service, like its SLOs for example, are organized under the name of the service. The following service types are supported:
APP_ENGINE
CLOUD_ENDPOINTS
ISTIO_CANONICAL_SERVICE
CLUSTER_ISTIO
CLOUD_RUN
GKE_SERVICE
GKE_NAMESPACE
GKE_WORKLOAD
CUSTOM
For more information, see Basic service types or Basic GKE service types.
You can add SLOs to any service in your project by using the API. Managing SLOs covers commands to manipulate SLOs.
Service IDsEach service has a fully qualified identifier called the resource name. This identifier is stored in the name
field of the service description, for example:
"name": "projects/PROJECT_NUMBER/services/PROJECT_ID-zone-us-central1-a-cloudrun-istio-system-cluster-local-gateway",
Embedded in the resource name is a shorter ID for the service, the part of the resource name after the substring projects/PROJECT_NUMBER/services/
If you created your own service with the resource name projects/PROJECT_NUMBER/services/my-test-service
, the service ID is my-test-service
.
For brevity and accuracy, many curl
examples assume the service ID is held in the environment variable SERVICE_ID
so you can refer to it repeatedly.
To retrieve information about all the services in your project, invoke the services.list
method. To retrieve information about a specific service by service ID, use the services.get
method:
curl
, invoke the services.list
or services.get
method by sending a GET
request to:
https://monitoring.googleapis.com/v3/projects/[PROJECT_ID]/services
to list all serviceshttps://monitoring.googleapis.com/v3/projects/[PROJECT_ID]/services/SERVICE_ID
to get a specific serviceFor example, the following request retrieves information about the service identified by the value stored in the environment variable SERVICE_ID
:
curl --http1.1 --header "Authorization: Bearer ${ACCESS_TOKEN}" https://monitoring.googleapis.com/v3/projects/${PROJECT_ID}/services/${SERVICE_ID}
Creating a service
To create a service, you specify a representation of the service type and send it to the services.create
method. You can the service-type structures described in Basic service types and Basic GKE service types.
The structures vary, but the process for calling the API is the same. You must specify a display name for the service and a basicService
field holding the service representation.
You can optionally specify the service ID you want the service to have. The following example shows the creation of a {[gke_name_short}} workload service:
ProtocolTo create the service by using curl
, send a POST
message to the https://monitoring.googleapis.com/v3/projects/[PROJECT_ID]/services
endpoint:
If you want to provide an ID for the service, create a variable for the service ID:
SERVICE_ID=my-test-service
This value is supplied in the URL query parameter service_id
.
Create a variable to hold the request body that describes the service:
CREATE_SERVICE_POST_BODY=$(cat <<EOF
{
"displayName": "My Test GKE Workload Service",
"basicService": {
"serviceType": "GKE_WORKLOAD",
"serviceLabels": {
"cluster_name": "workload-test",
"location": "us-central1-c",
"namespace_name": "kube-system",
"project_id": "PROJECT_ID",
"top_level_controller_name": "gke-metrics-controller",
"top_level_controller_type": "DaemonSet"
}
}
}
EOF
)
Post the request body to the endpoint, and specify the service ID as the value of the service_id
query parameter:
curl --http1.1 --header "Authorization: Bearer ${ACCESS_TOKEN}" --header "Content-Type: application/json" -X POST -d "${CREATE_SERVICE_POST_BODY}" https://monitoring.googleapis.com/v3/projects/${PROJECT_ID}/services?service_id=${SERVICE_ID}
For examples of the structures associated with other services, see Basic service types or Basic GKE service types.
Deleting a serviceTo delete a service you created, invoke the services.delete
method and specify the service ID.
curl
, send a DELETE
request to the https://monitoring.googleapis.com/v3/projects/[PROJECT_ID]/services/SERVICE_ID
endpoint:
curl --http1.1 --header "Authorization: Bearer ${ACCESS_TOKEN}" -X DELETE https://monitoring.googleapis.com/v3/projects/${PROJECT_ID}/services/${SERVICE_ID}
Managing SLOs
You can create, delete, and retrieve SLOs by using the SLO API. You can have up to 500 SLOs for each service.
To manage SLOs for a service, you must have the service ID. SLOs are named based on the service they belong to. For information about identifying service IDs, see Service IDs.
Listing SLOsTo retrieve information about all the SLOs associated with a service, invoke the services.serviceLevelObjectives.list
method. To retrieve information about a specific SLO by name, use the services.serviceLevelObjectives.get
method:
curl
, invoke the services.serviceLevelObjectives.list
or services.serviceLevelObjectives.get
method by sending a GET
request to:
https://monitoring.googleapis.com/v3/projects/[PROJECT_ID]/services/[SERVICE_ID]/serviceLevelObjectives
to list all SLOshttps://monitoring.googleapis.com/v3/projects/[PROJECT_ID]/services/[SERVICE_ID]/serviceLevelObjectives/SLO_ID
to get a specific SLOFor example, the following request lists all SLOs associated with the service ID stored in the environment variable SERVICE_ID
:
curl --http1.1 --header "Authorization: Bearer ${ACCESS_TOKEN}" https://monitoring.googleapis.com/v3/projects/${PROJECT_ID}/services/${SERVICE_ID}/serviceLevelObjectives
The following is an availability SLO retrieved from the “currencyservice” service:
{ "name": "projects/PROJECT_NUMBER/services/PROJECT_ID-zone-us-central1-c-csm-main-default-currencyservice/serviceLevelObjectives/3kavNVTtTMuzL7KcXAxqCQ", "displayName": "98% Availability in Calendar Week" "serviceLevelIndicator": { "basicSli": { "availability": {}, "location": [ "us-central1-c" ] } }, "goal": 0.98, "calendarPeriod": "WEEK", },
This SLO is built on an availability SLI, it sets a target performance goal of 98 percent, and it measures compliance over a calendar week. For more information on availability SLIs, see Service-level indicators.
See ServiceLevelObjective
for more information about the structure of SLOs.
Each SLO has a unique identifier within the service, consisting of the string following serviceLevelObjectives
in the SLO's name
field. In the following example:
"name": "projects/PROJECT_NUMBER/services/PROJECT_ID-zone-us-central1-c-csm-main-default-currencyservice/serviceLevelObjectives/3kavNVTtTMuzL7KcXAxqCQ",
the SLO ID is the string 3kavNVTtTMuzL7KcXAxqCQ
.
To retrieve information about this particular SLO, request the SLO by ID.
Protocol To get a specific SLO by usingcurl
, invoke the services.serviceLevelObjectives.get
method by sending a GET
request to https://monitoring.googleapis.com/v3/projects/[PROJECT_ID]/services/SERVICE_ID/serviceLevelObjectives/SLO_ID
:
SLO_ID=dhefHDM4TwSRZEKIV4ZYEg curl --http1.1 --header "Authorization: Bearer ${ACCESS_TOKEN}" https://monitoring.googleapis.com/v3/projects/${PROJECT_ID}/services/${SERVICE_ID}/serviceLevelObjectives/${SLO_ID}Creating an SLO
To create an SLO by using the SLO API, you must create a ServiceLevelObjective
object and pass it to the serviceLevelObjectives.create
method.
The structure of an SLO has many embedded structures, including one for the value of the serviceLevelIndicator
field.
For Cloud Service Mesh, Istio on Google Kubernetes Engine, and App Engine services, the SLIs are pre-defined. You can use the Cloud Service Mesh console to create SLOs; all you have to do is specify the performance goals and compliance periods.
For other services, you must define SLOs by using the SLO API. To specify an SLO, you must create a value for the serviceLevelIndicator
field, as well. See Creating a service-level indicator for some techniques, and Creating SLIs from metrics for a set of examples based on various sources.
You can also create SLIs by using the Google Cloud console. For more information, see Creating an SLO.
Skeleton of a SLOThe basic skeleton for building the SLO is as follows:
{ "displayName": string, "serviceLevelIndicator": { object (ServiceLevelIndicator) }, "goal": number, // Union field period can be only one of the following: "rollingPeriod": string, "calendarPeriod": enum (CalendarPeriod) // End of list of possible types for union field period. }
You must specify the following:
For more information about SLIs, performance goals, and compliance periods, see Concepts in service monitoring.
For Cloud Service Mesh, Istio on Google Kubernetes Engine, and App Engine services, the SLI type is the basic SLI.
For other services, you have to create a request-based SLI or a windows-based SLI. See Creating a service-level indicator for some techniques.
ExampleAfter you have the SLI, you can build the SLO. The following example defines an SLO for a service that uses a basic SLI. You might have several SLOs on a single SLI, for example, one for 95% availability and one for 99% availability. The following example is an SLO for 95% availability over a calendar week:
{ "serviceLevelIndicator": { "basicSli": { "availability": {}, "location": [ "us-central1-c" ] } }, "goal": 0.95, "calendarPeriod": "WEEK", "displayName": "95% Availability in Calendar Week" }
This example specifies an SLO for 75% availability over a rolling 3-day period:
{ "serviceLevelIndicator": { "basicSli": { "availability": {}, "location": [ "us-central1-c" ] } }, "goal": 0.75, "rollingPeriod": "259200s", "displayName": "75% Availability in Rolling 3 Days" }Protocol To create an SLO by using
curl
, send a POST
message to the https://monitoring.googleapis.com/v3/projects/[PROJECT_ID]/services/SERVICE_ID/serviceLevelObjectives
endpoint:
Create a variable for the service ID:
SERVICE_ID=custom:my-test-service
Create a variable to hold the request body, an instance of the ServiceLevelObjective
object. This example uses one of the SLOs described previously:
CREATE_SLO_POST_BODY=$(cat <<EOF
{
"serviceLevelIndicator": {
"basicSli": {
"availability": {},
"location": [
"us-central1-c"
]
}
},
"goal": 0.75,
"rollingPeriod": "259200s",
"displayName": "75% Availability in Rolling 3 Days"
}
EOF
)
Post the request body to the endpoint:
curl --http1.1 --header "Authorization: Bearer ${ACCESS_TOKEN}" --header "Content-Type: application/json" -X POST -d "${CREATE_SLO_POST_BODY}" https://monitoring.googleapis.com/v3/projects/${PROJECT_ID}/services/${SERVICE_ID}/serviceLevelObjectives
Optionally, you can also specify a desired ID for the SLO as the value of the service_level_objective_id
query parameter:
SLO_ID=test-slo-id
curl --http1.1 --header "Authorization: Bearer ${ACCESS_TOKEN}" --header "Content-Type: application/json" -X POST -d "${CREATE_SLO_POST_BODY}" https://monitoring.googleapis.com/v3/projects/${PROJECT_ID}/services/${SERVICE_ID}/serviceLevelObjectives?service_level_objective_id=${SLO_ID}
To delete an SLO, invoke the services.serviceLevelObjectives.delete
method and specify the ID of the SLO in your project:
curl
, send a DELETE
request to the https://monitoring.googleapis.com/v3/projects/[PROJECT_ID]/services/SERVICE_ID/serviceLevelObjectives/SLO_ID
endpoint:
curl --http1.1 --header "Authorization: Bearer ${ACCESS_TOKEN}" -X DELETE https://monitoring.googleapis.com/v3/projects/${PROJECT_ID}/services/${SERVICE_ID}/serviceLevelObjectives/${SLO_ID}
Accessing SLO time series
SLO data is stored in time series, so you can use the timeSeries.list
method to retrieve it. However, this data isn't stored in standard metric types, so you can't use the standard mechanism of specifying a metric-type filter to the timeSeries.list
method.
Instead, SLO time series are retrieved by specifying another type of filter, a time-series selector, to the timeSeries.list
method in the filter
parameter. See Retrieving SLO data for information on using these selectors.
You also use time-series selectors to set up alerting policies programmatically. See Alerting on your burn rate for more information.
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4