This page shows you how to enable the multi-cluster GKE Gateway controller, a Google-hosted controller that provisions external and internal load balancers for your GKE clusters. To learn how to use Gateway resources for container load balancing, see Deploying Gateways or Deploying multi-cluster Gateways.
The multi-cluster GKE Gateway Controller installs the following multi-cluster GatewayClasses in your clusters:
gke-l7-global-external-managed-mc
for global external multi-cluster Gatewaysgke-l7-regional-external-managed-mc
for regional external multi-cluster Gatewaysgke-l7-rilb-mc
for regional internal multi-cluster Gatewaysgke-l7-gxlb-mc
for global external Classic multi-cluster GatewaysLearn more about the capabilities of the various GatewayClasses in GKE.
PricingAll Compute Engine resources deployed through the Gateway controllers are charged against the project in which your GKE clusters reside. The single-cluster Gateway controller is offered at no additional charge as a part of GKE Standard and Autopilot pricing. Pricing for multi-cluster Gateways is described in the Multi Cluster Ingress and Gateway pricing page.
Before you beginBefore you start, make sure that you have performed the following tasks:
gcloud components update
. Note: For existing gcloud CLI installations, make sure to set the compute/region
property. If you use primarily zonal clusters, set the compute/zone
instead. By setting a default location, you can avoid errors in the gcloud CLI like the following: One of [--zone, --region] must be supplied: Please specify location
. You might need to specify the location in certain commands if the location of your cluster differs from the default that you set.HttpLoadBalancing
add-on enabled.Compute Network User
role to the GKE Service account for the service project.In addition to the GKE Gateway controller requirements, for multi-cluster Gateway deployments, make sure you have performed the following tasks:
Enable the following multi-cluster Gateways required APIs in your project:
Run this command to enable the required APIs if they are not already enabled:
gcloud services enable \
trafficdirector.googleapis.com \
multiclusterservicediscovery.googleapis.com \
multiclusteringress.googleapis.com \
--project=PROJECT_ID
Replace PROJECT_ID
with the project ID where your GKE clusters are running.
The same limitations and known issues for single-cluster Gateways also apply to multi-cluster Gateways.
In addition to the single-cluster Gateway limitations, the following limitations apply to multi-cluster Gateways:
Load balancing to backends in different regions is not supported with the regional internal GatewayClass gke-l7-rilb-mc
. To learn more about the different features supported with each GatewayClass, see GatewayClass capabilities.
A Service
is not supported as a backendRefs
with multi-cluster Gateway. Multi-cluster Gateway only supports ServiceImport
as a valid backendRefs
.
All clusters in the fleet must exist in the fleet host project.
Cross-project load balancing is not supported. All clusters (config cluster and target clusters) attached to the same multi-cluster Gateway must be deployed in the same Shared VPC host project or service project. For more information on supported Shared VPC topologies with multi-cluster Gateway, see Use multi-cluster Gateway with Shared VPC.
Cross-VPC load balancing is not supported. All clusters (config cluster and target clusters) attached to the same multi-cluster Gateway must be deployed in the same VPC.
Multi-cluster Gateway depends on MCS for handling cross-cluster service discovery. As such, services multi-cluster Gateway exposes are subject to all the Multi-cluster Service requirements.
Multi-cluster Gateways can leak load balancer resources under the following scenarios:
Gateway
resources which exist on the current config cluster.Gateway
resources which reference a multi-cluster GatewayClass
exist on the config cluster.Multi-cluster Gateway runs as a global service. If the multi-cluster Gateway controller encounters a regional Fleet (Hub) control plane failure, it responds by failing statically and making no further load balancer changes until the region returns to service.
GKE Gateway uses Cloud Load Balancing quotas to limit the number of resources that the Gateway controller can create to manage ingress traffic routed to GKE clusters.
Set up your environment for multi-cluster GatewaysIt requires multiple GKE clusters to complete the examples in Deploying multi-cluster Gateways. All of the clusters are registered to the same fleet so that multi-cluster Gateways and Services can operate across them.
The following steps will deploy three GKE clusters across two different regions in your project:
us-west1-a/gke-west-1
us-west1-a/gke-west-2
us-east1-b/gke-east-1
This will create the following cluster topology:
These GKE clusters are used to demonstrate multi-region load balancing and blue-green, multi-cluster traffic splitting using external and internal Gateways.
Deploy clustersIn these steps you will deploy three GKE clusters into regions us-east1
and us-west1
.
Clusters will be registered to your project's fleet. Grouping your GKE clusters together in a fleet allows them to be targeted by a multi-cluster Gateway.
Create a GKE cluster in us-west1
named gke-west-1
:
gcloud container clusters create gke-west-1 \
--gateway-api=standard \
--location=us-west1-a \
--workload-pool=PROJECT_ID.svc.id.goog \
--cluster-version=VERSION \
--enable-fleet \
--project=PROJECT_ID
Replace the following:
PROJECT_ID
: the project ID where your GKE clusters are running.VERSION
: the GKE version, 1.24 or later.Create another GKE cluster in us-west1
(or the same region as the previous cluster) named gke-west-2
:
gcloud container clusters create gke-west-2 \
--gateway-api=standard \
--location=us-west1-a \
--workload-pool=PROJECT_ID.svc.id.goog \
--cluster-version=VERSION \
--enable-fleet \
--project=PROJECT_ID
Create a GKE cluster in us-east1
(or a region that is different than the previous one) named gke-east-1
gcloud container clusters create gke-east-1 \
--gateway-api=standard \
--location=us-east1-b \
--workload-pool=PROJECT_ID.svc.id.goog \
--cluster-version=VERSION \
--enable-fleet \
--project=PROJECT_ID
Confirm that the clusters have been successfully registered to the fleet:
gcloud container fleet memberships list --project=PROJECT_ID
The output will be similar to the following:
NAME EXTERNAL_ID LOCATION
gke-east-1 45a80b37-4b00-49aa-a68b-b430fce1e3f0 us-east1
gke-west-2 ac7087a5-f5ee-401e-b430-57f3af141239 us-west1
gke-west-1 549efe3a-b18e-4eb9-8796-e50b7967cde2 us-west1
This step configures cluster credentials with memorable names. This makes it easier to switch between clusters when deploying resources across several clusters.
Fetch the credentials for cluster gke-west-1
, gke-west-2
, and gke-east-1
:
gcloud container clusters get-credentials gke-west-1 --location=us-west1-a --project=PROJECT_ID
gcloud container clusters get-credentials gke-west-2 --location=us-west1-a --project=PROJECT_ID
gcloud container clusters get-credentials gke-east-1 --location=us-east1-b --project=PROJECT_ID
This stores the credentials locally so that you can use your kubectl client to access the cluster API servers. By default an auto-generated name is created for the credential.
Rename the cluster contexts so they are easier to reference later:
kubectl config rename-context gke_PROJECT_ID_us-west1-a_gke-west-1 gke-west-1
kubectl config rename-context gke_PROJECT_ID_us-west1-a_gke-west-2 gke-west-2
kubectl config rename-context gke_PROJECT_ID_us-east1-b_gke-east-1 gke-east-1
Replace PROJECT_ID
with the project ID where your clusters are deployed.
Enable multi-cluster Services in your fleet for the registered clusters. This enables the MCS controller for the three clusters that are registered to your fleet so that it can start listening to and exporting Services.
gcloud container fleet multi-cluster-services enable \
--project PROJECT_ID
Grant Identity and Access Management (IAM) permissions required by the MCS controller:
gcloud projects add-iam-policy-binding PROJECT_ID \
--member "serviceAccount:PROJECT_ID.svc.id.goog[gke-mcs/gke-mcs-importer]" \
--role "roles/compute.networkViewer" \
--project=PROJECT_ID
Replace PROJECT_ID
with the project ID where your clusters are deployed.
Confirm that MCS is enabled for the registered clusters. You will see the memberships for the three registered clusters. It may take several minutes for all of the clusters to show.
gcloud container fleet multi-cluster-services describe --project=PROJECT_ID
The output is similar to the following:
createTime: '2023-10-12T06:14:33.466903587Z'
membershipStates:
projects/441323991697/locations/us-east1/memberships/gke-east-1:
state:
code: OK
description: Firewall successfully updated
updateTime: '2023-10-12T06:15:28.395318091Z'
projects/441323991697/locations/us-west1/memberships/gke-west-1:
state:
code: OK
description: Firewall successfully updated
updateTime: '2023-10-12T06:15:30.534594027Z'
projects/441323991697/locations/us-west1/memberships/gke-west-2:
state:
code: OK
description: Firewall successfully updated
updateTime: '2023-10-12T06:15:29.110582109Z'
name: projects/pierre-louis-playground/locations/global/features/multiclusterservicediscovery
resourceState:
state: ACTIVE
spec: {}
updateTime: '2023-10-12T06:15:31.027276757Z'
The multi-cluster GKE Gateway controller governs the deployment of multi-cluster Gateways.
Note: the multi-cluster Gateway controller is also responsible for the instrumentation ofMulticlusterIngress
resources.
When enabling the multi-cluster Gateway controller, you must select your config cluster. The config cluster is the GKE cluster in which your Gateway resources (Gateway, Routes, Policies) are deployed. It is a central place that controls routing across your clusters. See Config cluster design to help you decide which cluster to choose as your config cluster.
Enable multi-cluster Gateway and specify your config cluster in your fleet. Note that you can always update the config cluster at a later time. This example specifies gke-west-1
as the config cluster that will host the resources for multi-cluster Gateways.
gcloud container fleet ingress enable \
--config-membership=projects/PROJECT_ID/locations/us-west1/memberships/gke-west-1 \
--project=PROJECT_ID
Grant Identity and Access Management (IAM) permissions required by the multi-cluster Gateway controller:
gcloud projects add-iam-policy-binding PROJECT_ID \
--member "serviceAccount:service-PROJECT_NUMBER@gcp-sa-multiclusteringress.iam.gserviceaccount.com" \
--role "roles/container.admin" \
--project=PROJECT_ID
Replace PROJECT_ID
and PROJECT_NUMBER
with the project ID and project number where your clusters are deployed.
Confirm that the GKE Gateway controller is enabled for your fleet:
gcloud container fleet ingress describe --project=PROJECT_ID
The output is similar to the following:
createTime: '2023-10-12T06:23:06.732858524Z'
membershipStates:
projects/441323991697/locations/us-east1/memberships/gke-east-1:
state:
code: OK
updateTime: '2023-10-12T06:30:08.815839024Z'
projects/441323991697/locations/us-west1/memberships/gke-west-1:
state:
code: OK
updateTime: '2023-10-12T06:30:08.815837031Z'
projects/441323991697/locations/us-west1/memberships/gke-west-2:
state:
code: OK
updateTime: '2023-10-12T06:30:08.815840985Z'
name: projects/pierre-louis-playground/locations/global/features/multiclusteringress
resourceState:
state: ACTIVE
spec:
multiclusteringress:
configMembership: projects/pierre-louis-playground/locations/us-west1/memberships/gke-west-1
state:
state:
code: OK
description: Ready to use
updateTime: '2023-10-12T06:23:51.317464415Z'
updateTime: '2023-10-12T06:30:09.439319551Z'
Confirm that the GatewayClasses exist in your config cluster:
kubectl get gatewayclasses --context=gke-west-1
The output is similar to the following:
NAME CONTROLLER ACCEPTED AGE
gke-l7-global-external-managed networking.gke.io/gateway True 78m
gke-l7-global-external-managed-mc networking.gke.io/gateway True 4m22s
gke-l7-gxlb networking.gke.io/gateway True 78m
gke-l7-gxlb-mc networking.gke.io/gateway True 4m23s
gke-l7-regional-external-managed networking.gke.io/gateway True 78m
gke-l7-regional-external-managed-mc networking.gke.io/gateway True 4m22s
gke-l7-rilb networking.gke.io/gateway True 78m
gke-l7-rilb-mc networking.gke.io/gateway True 4m22s
This output includes the GatewayClass gke-l7-global-external-managed-mc, gke-l7-regional-external-managed-mc, gke-l7-gxlb-mc for external multi-cluster Gateways and the GatewayClass gke-l7-rilb-mc for internal multi-cluster Gateways.
Switch your kubectl context to the config cluster:
kubectl config use-context gke-west-1
You are now ready to begin deploying multi-cluster Gateways in the config cluster.
TroubleshootingThis section shows you how to resolve issues related to the multi-cluster Gateway controller enablement.
GatewayClasses are not available in the config clusterThe following error might occur when you run the command kubectl get gatewayclasses
:
error: the server doesn't have a resource type "gatewayclasses"
To resolve this issue, install the Gateway API on your cluster:
gcloud container clusters update CLUSTER_NAME \
--gateway-api=standard \
--location=CONTROL_PLANE_LOCATION
Replace the following:
CLUSTER_NAME
: the name of your cluster.CONTROL_PLANE_LOCATION
: the Compute Engine location of the control plane of your cluster. Provide a region for regional clusters, or a zone for zonal clusters.When you enable the Gateway API on the config cluster and include the CRDs after the Fleet ingress feature is enabled, the multi-cluster Gateway controller might fail to start.
You can avoid this issue by enabling the Gateway API before you enable Fleet ingress. However, if you've already enabled Fleet ingress, resolve this issue by disabling and then re-enabling Fleet ingress:
Disable Fleet ingress:
gcloud container fleet ingress disable
Enable Fleet ingress:
gcloud container fleet ingress enable \
--config-membership=CONFIG_MEMBERSHIP \
--project=PROJECT_ID
Replace the following:
projects/PROJECT_ID/locations/us-west1/memberships/gke-west-1
.RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4