Stay organized with collections Save and categorize content based on your preferences.
This page shows you how to resolve issues related to application-layer secrets encryption in Google Kubernetes Engine (GKE).
Failed updateWhen you update the encryption configuration of application-layer secrets encryption, GKE must rewrite all Secret objects in the Kubernetes cluster. GKE does this to ensure that all Secrets are encrypted by the new Cloud KMS Key, or are written un-encrypted if that is what you configure.
This update operation can fail due to any of the following conditions:
AdmissionWebhook
prevents GKE from being able to update Secret objects.Until the update operation is successful, don't interact with either the updated or previous Cloud KMS keys.
Debugging fieldsNew GKE clusters running version 1.29 and later contain additional fields that help you track updates to Cluster.DatabaseEncryption
and help you recover from failures.
The following steps only apply to clusters where the DatabaseEncryption.CurrentState
field is not empty. If the CurrentState
field is empty, the feature is not enabled on this cluster version yet.
The following limits apply to these fields:
CurrentState
field
You can inspect the current status of a DatabaseEncryption
update operation by examining the CurrentState
field in Cluster.DatabaseEncryption
.
CurrentState
Description
CURRENT_STATE_ENCRYPTED
CURRENT_STATE_DECRYPTED
CURRENT_STATE_ENCRYPTION_PENDING
CURRENT_STATE_DECRYPTION_PENDING
CURRENT_STATE_ENCRYPTION_ERROR
CURRENT_STATE_DECRYPTION_ERROR
There was an error with the most recent update. Don't disable or destroy any previously used Cloud KMS keys, as they might still be in use by GKE.
Refer to the LastOperationErrors
field for more information.
LastOperationErrors
field
When an update operation fails, the underlying error from the GKE control plane is displayed in the output of gcloud container clusters update
.
The error messages from the two most recent failed update operations are also available in Cluster.DatabaseEncryption.LastOperationErrors
.
DecryptionKeys
field
The Cloud KMS key used for new encryption operations is shown in DatabaseEncryption.KeyName
. Usually this is the only key used by the cluster.
However, DatabaseEncryption.DecryptionKeys
contains additional keys that are also used by the cluster if an update is in progress or after a failure.
To recover from a failed update, do the following:
gcloud container clusters update ... --database-encryption-key
. We recommend that you retry with the same update request that you originally issued, or update the cluster back to the previous state. GKE might not be able to transition to a different key or encryption state if it can't read one or more Secrets.The following sections list common reasons for errors.
Cloud KMS key errorIf the error message contains a reference to one or more Cloud KMS keys, examine your Cloud KMS key configuration to make sure the relevant key versions are usable.
If the error indicates that a Cloud KMS key has been disabled or destroyed, re-enable the key or key version.
Error: Unable to use CloudKMS key configured for Application Level encryptionThe following error message occurs if GKE's default service account can't access the Cloud KMS key:
Cluster problem detected (Kubernetes Engine Service Agent account unable to use CloudKMS key configured for Application Level encryption).
To resolve this issue, re-enable the disabled key.
Unable to update SecretThe following error might occur if the Kubernetes API rejected the update request due to an admission webhook:
error admission webhook WEBHOOK_NAME denied the request
To resolve this error, remove the webhook or modify it so that GKE can update Secrets in all namespaces during key updates.
Error: the namespace is managedThe following error occurs when you try to re-encrypt Secrets that are in a GKE-managed namespace, such as kube-system
, in an Autopilot cluster. The error message is similar to the following:
Error from server (Forbidden): secrets "alertmanager" is
forbidden: User cannot patch resource "secrets" in API group "" in the namespace "gke-gmp-system":
GKE Warden authz [denied by managed-namespaces-limitation]: the namespace "gke-gmp-system"
is managed and the request's verb "patch" is denied'
GKE Autopilot clusters don't allow you to modify Kubernetes resources, including Secrets, in managed namespaces.
To resolve this error, do the following:
kubectl
commands to re-encrypt Secrets, use the --namespace
flag to scope the command to namespaces that you manage.If you can't find a solution to your problem in the documentation, see Get support for further help, including advice on the following topics:
google-kubernetes-engine
tag to search for similar issues. You can also join the #kubernetes-engine
Slack channel for more community support.Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see the Google Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.
Last updated 2025-08-12 UTC.
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Hard to understand","hardToUnderstand","thumb-down"],["Incorrect information or sample code","incorrectInformationOrSampleCode","thumb-down"],["Missing the information/samples I need","missingTheInformationSamplesINeed","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2025-08-12 UTC."],[],[]]
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4