This page shows you how to add, manage, scale, upgrade, and delete node pools that run in your Google Kubernetes Engine (GKE) Standard clusters. You also learn how to deploy Pods to specific node pools, and about the implications of node pool upgrades on running workloads.
The information in this page helps you optimize your GKE Standard clusters for performance and scalability.
This page is for Operators, Cloud architects, and Developers, who need to create and configure clusters, and deploy workloads on GKE. To learn more about common roles and example tasks that we reference in Google Cloud content, see Common GKE user roles and tasks.
Before reading this page, ensure that you're familiar with node pools.
Before you beginBefore you start, make sure that you have performed the following tasks:
gcloud components update
. Note: For existing gcloud CLI installations, make sure to set the compute/region
property. If you use primarily zonal clusters, set the compute/zone
instead. By setting a default location, you can avoid errors in the gcloud CLI like the following: One of [--zone, --region] must be supplied: Please specify location
. You might need to specify the location in certain commands if the location of your cluster differs from the default that you set.GKE uses IAM service accounts that are attached to your nodes to run system tasks like logging and monitoring. At a minimum, these node service accounts must have the Kubernetes Engine Default Node Service Account (roles/container.defaultNodeServiceAccount
) role on your project. By default, GKE uses the Compute Engine default service account, which is automatically created in your project, as the node service account.
To grant the roles/container.defaultNodeServiceAccount
role to the Compute Engine default service account, complete the following steps:
PROJECT_NUMBER-compute@developer.gserviceaccount.comReplace
PROJECT_NUMBER
with the project number that you copied.gcloud projects describe PROJECT_ID \ --format="value(projectNumber)"
Replace PROJECT_ID
with your project ID.
The output is similar to the following:
12345678901
roles/container.defaultNodeServiceAccount
role to the Compute Engine default service account:
gcloud projects add-iam-policy-binding PROJECT_ID \ --member="serviceAccount:PROJECT_NUMBER-compute@developer.gserviceaccount.com" \ --role="roles/container.defaultNodeServiceAccount"
Replace PROJECT_NUMBER
with the project number from the previous step.
You can add a new node pool to a GKE Standard cluster using the gcloud CLI, the Google Cloud console, or Terraform. GKE also supports node auto-provisioning, which automatically manages the node pools in your cluster based on scaling requirements.
Best practice:Create and use a minimally-privileged Identity and Access Management (IAM) service account for your node pools to use instead of the Compute Engine default service account. For instructions to create a minimally-privileged service account, refer to Hardening your cluster's security.
gcloudTo create a node pool, run the gcloud container node-pools create
command:
gcloud container node-pools create POOL_NAME \
--cluster CLUSTER_NAME \
--service-account SERVICE_ACCOUNT
Replace the following:
POOL_NAME
: the name of the new node pool.CLUSTER_NAME
: the name of your existing cluster.SERVICE_ACCOUNT
: the name of the IAM service account for your nodes to use.
We strongly recommend that you specify a minimally-privileged IAM service account that your nodes can use instead of the Compute Engine default service account. To learn how to create a minimally-privileged service account, see Use a least privilege service account.
To specify a custom service account in the gcloud CLI, add the following flag to your command:
--service-account=SERVICE_ACCOUNT_NAME@PROJECT_ID.iam.gserviceaccount.com
Replace SERVICE_ACCOUNT_NAME with the name of your minimally-privileged service account.
For a full list of optional flags you can specify, refer to the gcloud container node-pools create
documentation.
The output is similar to the following:
Creating node pool POOL_NAME...done.
Created [https://container.googleapis.com/v1/projects/PROJECT_ID/zones/us-central1/clusters/CLUSTER_NAME/nodePools/POOL_NAME].
NAME: POOL_NAME
MACHINE_TYPE: e2-medium
DISK_SIZE_GB: 100
NODE_VERSION: 1.21.5-gke.1302
In this output, you see details about the node pool, such as the machine type and GKE version running on the nodes.
Occasionally, the node pool is created successfully but the gcloud
command times out instead of reporting the status from the server. To check the status of all node pools, including those not yet fully provisioned, use the following command:
gcloud container node-pools list --cluster CLUSTER_NAME
Console
To add a node pool to an existing Standard cluster, perform the following steps:
Go to the Google Kubernetes Engine page in Google Cloud console.
In the cluster list, click the name of the Standard cluster you want to modify.
Click add_box Add node pool.
Configure your node pool.
In the navigation menu, click Security.
We strongly recommend that you specify a minimally-privileged IAM service account that your nodes can use instead of the Compute Engine default service account. To learn how to create a minimally-privileged service account, see Use a least privilege service account.
Click Create to add the node pool.
Use one of the following examples:
Create an IAM service account and grant it the roles/container.defaultNodeServiceAccount
role on the project:
Create a node pool that uses the new service account:
To learn more about using Terraform, see Terraform support for GKE.
View node pools in a Standard cluster gcloudTo list all the node pools of a Standard cluster, run the gcloud container node-pools list
command:
gcloud container node-pools list --cluster CLUSTER_NAME
To view details about a specific node pool, run the gcloud container node-pools describe
command:
gcloud container node-pools describe POOL_NAME \
--cluster CLUSTER_NAME
Replace the following:
CLUSTER_NAME
: the name of the cluster.POOL_NAME
: the name of the node pool to view.To view node pools for a Standard cluster, perform the following steps:
Go to the Google Kubernetes Engine page in Google Cloud console.
In the cluster list, click the name of the Standard cluster.
Click the Nodes tab.
Under Node Pools, click the name of the node pool you want to view.
You can scale your node pools up or down to optimize for performance and cost. With GKE Standard node pools, you can scale a node pool horizontally by changing the number of nodes in the node pool, or scale a node pool vertically by changing the machine attribute configuration of the nodes.
Horizontally scale by changing the node count gcloudTo resize a cluster's node pools, run the gcloud container clusters resize
command:
gcloud container clusters resize CLUSTER_NAME \
--node-pool POOL_NAME \
--num-nodes NUM_NODES
Replace the following:
CLUSTER_NAME
: the name of the cluster to resize.POOL_NAME
: the name of the node pool to resize.NUM_NODES
: the number of nodes in the pool in a zonal cluster. If you use multi-zonal or regional clusters, NUM_NODES is the number of nodes for each zone the node pool is in.Repeat this command for each node pool. If your cluster has only one node pool, omit the --node-pool
flag.
To resize a cluster's node pools, perform the following steps:
Go to the Google Kubernetes Engine page in Google Cloud console.
In the cluster list, click the name of the Standard cluster you want to modify.
Click the Nodes tab.
In the Node Pools section, click the name of the node pool that you want to resize.
Click edit Resize.
In the Number of nodes field, enter how many nodes that you want in the node pool, and then click Resize.
Repeat for each node pool as needed.
You can modify the node pool's configured machine type, disk type, and disk size.
When you edit one or more of these machine attributes, GKE updates the nodes to the new configuration using the upgrade strategy configured for the node pool. If you configure the blue-green upgrade strategy you can migrate the workloads from the original nodes to the new nodes while being able to roll back the original nodes if the migration fails. Inspect the upgrade settings of the node pool to ensure that the configured strategy is how you want your nodes to be updated.
Update at least one of the highlighted machine attributes in the following command:
gcloud container node-pools update POOL_NAME \
--cluster CLUSTER_NAME \
--machine-type MACHINE_TYPE \
--disk-type DISK_TYPE \
--disk-size DISK_SIZE
Omit any flags for machine attributes that you don't want to change. However, you must use at least one machine attribute flag, as the command otherwise fails.
Replace the following:
POOL_NAME
: the name of the node pool to resize.CLUSTER_NAME
: the name of the cluster to resize.MACHINE_TYPE
: the type of machine to use for nodes. To learn more, see gcloud container node-pools update.DISK_TYPE
: the type of the node VM boot disk, must be one of pd-standard
, pd-ssd
, pd-balanced
.DISK_SIZE
: the size for node VM boot disks in GB. Defaults to 100GB.This change requires recreating the nodes, which can cause disruption to your running workloads. For details about this specific change, find the corresponding row in the manual changes that recreate the nodes using a node upgrade strategy without respecting maintenance policies table. To learn more about node updates, see Planning for node update disruptions.
Caution: GKE immediately begins recreating the nodes for this change using the node upgrade strategy, regardless of active maintenance policies. GKE depends on resource availability for the change. Disabling node auto-upgrades doesn't prevent this change. Ensure that your workloads running on the nodes are prepared for disruption before you initiate this change. Upgrade a node poolBy default, a cluster's nodes have auto-upgrade enabled. Node auto-upgrades ensure that your cluster's control plane and node version remain in sync and in compliance with the Kubernetes version skew policy, which ensures that control planes are compatible with nodes up to two minor versions older than the control plane. For example, Kubernetes 1.29 control planes are compatible with Kubernetes 1.27 nodes.
Best practice:Avoid disabling node auto-upgrades so that your cluster benefits from the upgrades listed in the preceding paragraph.
With GKE node pool upgrades, you can choose between two configurable upgrade strategies, namely surge upgrades and blue-green upgrades.
Choose a strategy and use the parameters to tune the strategy to best fit your cluster environment's needs.
How node upgrades workWhile a node is being upgraded, GKE stops scheduling new Pods onto it, and attempts to schedule its running Pods onto other nodes. This is similar to other events that re-create the node, such as enabling or disabling a feature on the node pool.
During automatic or manual node upgrades, PodDisruptionBudgets (PDBs) and Pod termination grace period are respected for a maximum of 1 hour. If Pods running on the node can't be scheduled onto new nodes after one hour, GKE initiates the upgrade anyway. This behavior applies even if you configure your PDBs to always have all of your replicas available by setting the maxUnavailable
field to 0
or 0%
or by setting the minAvailable
field to 100%
or to the number of replicas. In all of these scenarios, GKE deletes the Pods after one hour so that the node deletion can happen.
If a workload requires more flexibility with graceful termination, use blue-green upgrades which provide settings for additional soak time to extend PDB checks beyond the one hour default.
To learn more about what to expect during node termination in general, see the topic about Pods.
The upgrade is only complete when all nodes have been recreated and the cluster is in the desired state. When a newly-upgraded node registers with the control plane, GKE marks the node as schedulable.
New node instances run the desired Kubernetes version as well as:
For a node pool upgrade to be considered complete, all nodes in the node pool must be recreated. If an upgrade started but then didn't complete and is in a partially upgraded state, the node pool version might not reflect the version of all of the nodes. To learn more, see Some node versions don't match the node pool version after an incomplete node pool upgrade. To determine that the node pool upgrade finished, check the node pool upgrade status. If the upgrade operation is beyond the retention period, then check that each individual node version matches the node pool version.
Manually upgrade a node poolYou can manually upgrade a node pool version to match the version of the control plane or to a previous version that is still available and is compatible with the control plane. You can manually upgrade multiple node pools in parallel, whereas GKE automatically upgrades only one node pool at a time.
When you manually upgrade a node pool, GKE removes any labels you added to individual nodes using kubectl
. To avoid this, apply labels to node pools instead.
Before you manually upgrade your node pool, consider the following conditions:
kubectl get ing
command. If the instance group is not synced, you can work around the problem by re-applying the manifest used to create the ingress.You can manually upgrade your node pools to a version compatible with the control plane, using the Google Cloud console or the Google Cloud CLI.
gcloudThe following variables are used in the commands in this section:
CLUSTER_NAME
: the name of the cluster of the node pool to be upgraded.NODE_POOL_NAME
: the name of the node pool to be upgraded.CONTROL_PLANE_LOCATION
: the location (region or zone) for the control plane, such as us-central1
or us-central1-a
.VERSION
: the Kubernetes version to which the nodes are upgraded. For example, --cluster-version=1.7.2
or cluster-version=latest
.Upgrade a node pool:
gcloud container clusters upgrade CLUSTER_NAME \
--node-pool=NODE_POOL_NAME \
--location=CONTROL_PLANE_LOCATION
To specify a different version of GKE on nodes, use the optional --cluster-version
flag:
gcloud container clusters upgrade CLUSTER_NAME \
--node-pool=NODE_POOL_NAME \
--location=CONTROL_PLANE_LOCATION \
--cluster-version VERSION
For more information about specifying versions, see Versioning.
For more information, refer to the gcloud container clusters upgrade
documentation.
To upgrade a node pool using the Google Cloud console, perform the following steps:
Go to the Google Kubernetes Engine page in Google Cloud console.
Click the name of the cluster.
On the Cluster details page, click the Nodes tab.
In the Node Pools section, click the name of the node pool that you want to upgrade.
Click edit Edit.
Click Change under Node version.
Select the desired version from the Node version drop-down list, then click Change.
It may take several minutes for the node version to change.
Deploy a Pod to a specific node poolYou can explicitly deploy a Pod to a specific node pool by using a nodeSelector
in your Pod manifest. nodeSelector
schedules Pods onto nodes with a matching label.
All GKE node pools have labels with the following format: cloud.google.com/gke-nodepool: POOL_NAME
. Add this label to the nodeSelector
field in your Pod as shown in the following example:
apiVersion: v1
kind: Pod
metadata:
name: nginx
labels:
env: test
spec:
containers:
- name: nginx
image: nginx
imagePullPolicy: IfNotPresent
nodeSelector:
cloud.google.com/gke-nodepool: POOL_NAME
For more information, see Assigning Pods to Nodes.
As an alternative to node selector, you can use node affinity. Use node affinity if you want a "soft" rule where the Pod attempts to meet the constraint, but is still scheduled even if the constraint can't be satisfied. For more information, see Node affinity. You can also specify resource requests for the containers.
Downgrade a node poolYou can downgrade a node pool, for example, to mitigate an unsuccessful node pool upgrade. Review the limitations before downgrading a node pool.
Best practice:Use the blue-green node upgrade strategy if you need to optimize for risk mitigation for node pool upgrades impacting your workloads. With this strategy, you can roll backan in-progress upgrade to the original nodes if the upgrade is unsuccessful.
Deleting a node pool deletes the nodes and all running workloads, not respecting PodDisruptionBudget
settings. To learn more about how this affects your workloads, including interactions with node selectors, see Deleting node pools.
To delete a node pool, run the gcloud container node-pools delete
command:
gcloud container node-pools delete POOL_NAME \
--cluster CLUSTER_NAME
Console
To delete a node pool, perform the following steps:
Go to the Google Kubernetes Engine page in Google Cloud console.
In the cluster list, click the name of the Standard cluster you want to modify.
Click the Nodes tab.
In the Node Pools section, click delete next to the node pool you want to delete.
When prompted to confirm, click Delete.
To learn about different approaches for moving workloads between machine types, for example, to migrate to a newer machine type, see Migrate nodes to a different machine type.
Migrate workloads between node poolsTo migrate workloads from one node pool to another node pool, see Migrate workloads between node pools. For example, you can use these instructions if you're replacing an existing node pool with a new node pool and you want to ensure that the workloads move to the new nodes from the existing nodes.
TroubleshootFor troubleshooting information, see Troubleshoot Standard node pools and Troubleshoot node registration.
What's nextkubelet
and sysctl
using Node System Configuration.RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4