This page shows you how to deploy an external LoadBalancer Service that builds a backend service-based external passthrough Network Load Balancer. Before reading this page, ensure that you're familiar with the following:
To learn more about external passthrough Network Load Balancers in general, see Backend service-based external passthrough Network Load Balancer.
Before you beginBefore you start, make sure that you have performed the following tasks:
gcloud components update
. Note: For existing gcloud CLI installations, make sure to set the compute/region
property. If you use primarily zonal clusters, set the compute/zone
instead. By setting a default location, you can avoid errors in the gcloud CLI like the following: One of [--zone, --region] must be supplied: Please specify location
. You might need to specify the location in certain commands if the location of your cluster differs from the default that you set.The HttpLoadBalancing
add-on must be enabled in your cluster. This add-on is enabled by default. It allows the cluster to manage load balancers which use backend services.
To create an external LoadBalancer Service that uses a backend service-based External passthrough Network Load Balancer, your GKE cluster must use version 1.25.5 or later.
To create an external LoadBalancer Service that uses weighted load balancing, your GKE cluster must use version 1.31.0-gke.1506000 or later.
To create an external LoadBalancer Service that uses GCE_VM_IP
network endpoint group (NEG) backends, your GKE cluster must use version 1.32.2-gke.1652000 or later.
You can create a new cluster or choose an existing cluster that meets the requirements.
Create a new clusterAutopilot
To create a new Autopilot cluster:
gcloud container clusters create-auto CLUSTER_NAME \
--release-channel=RELEASE_CHANNEL \
--cluster-version=VERSION \
--location=COMPUTE_LOCATION
Replace the following:
CLUSTER_NAME
: the name of the new cluster.RELEASE_CHANNEL
: the name of the GKE release channel for the cluster.VERSION
: the GKE version for the cluster.COMPUTE_LOCATION
: the Compute Engine region of the cluster.To disable automatic VPC firewall rule creation for LoadBalancer Services, include the --disable-l4-lb-firewall-reconciliation
flag. For more information, see User-managed firewall rules for GKE LoadBalancer Services.
Standard
To create a new Standard cluster:
gcloud container clusters create CLUSTER_NAME \
--release-channel=RELEASE_CHANNEL \
--cluster-version=VERSION \
--location=COMPUTE_LOCATION
Replace the following:
CLUSTER_NAME
: the name of the new cluster.RELEASE_CHANNEL
: the name of the GKE release channel for the cluster.VERSION
: the GKE version for the cluster.COMPUTE_LOCATION
: the Compute Engine region of the cluster.To disable automatic VPC firewall rule creation for LoadBalancer Services, include the --disable-l4-lb-firewall-reconciliation
flag. For more information, see User-managed firewall rules for GKE LoadBalancer Services.
Use the gcloud CLI to update an existing cluster:
gcloud container clusters upgrade CLUSTER_NAME \
--cluster-version=VERSION \
--master \
--location=COMPUTE_LOCATION
Replace the following:
CLUSTER_NAME
: the name of the existing cluster.VERSION
: the specific GKE version to which you want to upgrade your cluster. For more information, see Manually upgrading the control plane.COMPUTE_LOCATION
: the Compute Engine region of the cluster.To disable automatic VPC firewall rule creation for LoadBalancer Services, include the --disable-l4-lb-firewall-reconciliation
flag. For more information, see User-managed firewall rules for GKE LoadBalancer Services.
Deploy the following sample workload which provides the serving Pods for the external LoadBalancer Service.
Save the following sample Deployment as store-deployment.yaml
:
apiVersion: apps/v1
kind: Deployment
metadata:
name: store
spec:
replicas: 20
selector:
matchLabels:
app: store
template:
metadata:
labels:
app: store
spec:
containers:
- image: gcr.io/google_containers/echoserver:1.10
imagePullPolicy: Always
name: echoserver
ports:
- name: http
containerPort: 8080
readinessProbe:
httpGet:
path: /healthz
port: 8080
scheme: HTTP
Apply the manifest to the cluster:
kubectl apply -f store-deployment.yaml
Verify that there are 20 serving Pods for the Deployment:
kubectl get pods
The output is similar to the following:
NAME READY STATUS RESTARTS AGE
store-cdb9bb4d6-s25vw 1/1 Running 0 10s
store-cdb9bb4d6-vck6s 1/1 Running 0 10s
....
Expose the sample workload by creating an external LoadBalancer Service.
Save the following Service manifest as store-v1-lb-svc.yaml
:
apiVersion: v1
kind: Service
metadata:
name: store-v1-lb-svc
annotations:
cloud.google.com/l4-rbs: "enabled"
spec:
type: LoadBalancer
selector:
app: store
ports:
- name: tcp-port
protocol: TCP
port: 8080
targetPort: 8080
Apply the manifest to the cluster:
kubectl apply -f store-v1-lb-svc.yaml
Note the following points about this sample manifest:
The Service manifest must include the cloud.google.com/l4-rbs: "enabled"
annotation at the time the manifest is first applied to the cluster. This instructs GKE to create a backend service-based external passthrough Network Load Balancer. Backend service-based external passthrough Network Load Balancers are required to support features like IPv6 and weighted load balancing.
GKE uses either GCE_VM_IP
NEG backends or unmanaged instance group backends, depending on the cluster's version. In clusters with version 1.32.2-gke.1652000, the backend service-based external passthrough Network Load Balancer uses GCE_VM_IP
NEGs. In previous versions, the backend service-based external passthrough Network Load Balancer uses unmanaged instance groups.
If you add the cloud.google.com/l4-rbs: "enabled"
annotation to the manifest of an existing external LoadBalancer Service (that is, after the load balancer has been created), GKE ignores the annotation. External LoadBalancer Services created without this annotation in their manifests use target pool-based external passthrough Network Load Balancers. Using target pool-based external passthrough Network Load Balancers is discouraged.
To distribute new connections proportionally to nodes based on how many serving, ready, and non-terminating Pods are present on each node, enable weighted load balancing by adding the networking.gke.io/weighted-load-balancing: "pods-per-node"
annotation to the Service manifest.
Add the networking.gke.io/weighted-load-balancing: "pods-per-node"
annotation to the store-v1-lb-svc.yaml
Service manifest, and ensure that you also set the externalTrafficPolicy: Local
so that it looks like this:
apiVersion: v1
kind: Service
metadata:
name: store-v1-lb-svc
annotations:
cloud.google.com/l4-rbs: "enabled"
networking.gke.io/weighted-load-balancing: "pods-per-node"
spec:
type: LoadBalancer
externalTrafficPolicy: Local
selector:
app: store
ports:
- name: tcp-port
protocol: TCP
port: 8080
targetPort: 8080
Apply the manifest to the cluster:
kubectl apply -f store-v1-lb-svc.yaml
Note the following about this example on weighted load balancing:
The Service manifest uses externalTrafficPolicy: Local
. If you don't need to enable weighted load balancing, you can also use externalTrafficPolicy: Cluster
. For details about how the externalTrafficPolicy
defines node grouping, which nodes pass their load balancer health checks, and how packets are processed, see LoadBalancer Service concepts.
If you enable weighted load balancing, GKE doesn't prevent you from using externalTrafficPolicy: Cluster
, but externalTrafficPolicy: Cluster
effectively disables weighted load balancing because the packet might be routed, after the load balancer, to a different node.
You can also enable weighted load balancing on an existing external LoadBalancer Service by using kubectl edit svc service-name
. The kubectl edit
command opens the existing load balancer's Service manifest in your configured text editor, where you can modify the manifest and save changes. When you edit an existing external LoadBalancer Service, note the following points:
The existing external LoadBalancer Service must have resulted in the creation of a backend service-based external passthrough Network Load Balancers. This means the existing external LoadBalancer Service must have included the cloud.google.com/l4-rbs: "enabled"
annotation when the manifest was first applied to the cluster.
Adding the networking.gke.io/weighted-load-balancing: "pods-per-node"
annotation to an existing external LoadBalancer Service that uses a target pool-based external passthrough Network Load Balancer has no effect.
When updating the existing external LoadBalancer Service manifest, make sure to set externalTrafficPolicy: Local
. Using externalTrafficPolicy: Cluster
effectively disables weighted load balancing because the packet might be routed, after the load balancer, to a different node.
To distribute new connections to nodes regardless of how many serving Pods are present on each node, disable weighted load balancing by removing the networking.gke.io/weighted-load-balancing: "pods-per-node"
annotation from the Service manifest.
Verify that your Service is running:
kubectl get svc store-v1-lb-svc
The output is similar to the following:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
store-v1-lb-svc LoadBalancer 10.44.196.160 35.193.28.231 8080:32466/TCP 11m
GKE assigned an EXTERNAL_IP
for the external passthrough Network Load Balancer.
Test connecting to the load balancer:
curl EXTERNAL_IP:PORT
Replace the following:
EXTERNAL_IP
: the allocated IP address for the external passthrough Network Load Balancer.PORT
: the allocated port number for the external passthrough Network Load Balancer.The output is similar to the following:
Hostname: store-v1-lb-svc-cdb9bb4d6-hflxd
Pod Information:
-no pod information available-
Server values:
server_version=nginx: 1.13.3 - lua: 10008
Request Information:
client_address=10.128.0.50
method=GET
real path=/
query=
request_version=1.1
request_scheme=http
request_uri=EXTERNAL_IP
Request Headers:
accept=*/*
host=EXTERNAL_IP
user-agent=curl/7.81.0
Request Body:
-no body in request-
Check your LoadBalancer Service and its set of annotations describing its Google Cloud resources:
kubectl describe svc store-v1-lb-svc
The output is similar to the following:
Name: my-service-external
Namespace: default
Labels: <none>
Annotations: cloud.google.com/l4-rbs: enabled
cloud.google.com/neg-status: {"network_endpoint_groups":{"0":"k8s2-qvveq1d8-default-my-service-ext-5s55db85"},"zones":["us-central1-c"]} #This annotation appears in the output only if the service uses NEG backends.
networking.gke.io/weighted-load-balancing: pods-per-node #This annotation appears in the output only if weighted load balancing is enabled.
service.kubernetes.io/backend-service: k8s2-qvveq1d8-default-my-service-ext-5s55db85
service.kubernetes.io/firewall-rule: k8s2-qvveq1d8-default-my-service-ext-5s55db85
service.kubernetes.io/firewall-rule-for-hc: k8s2-qvveq1d8-default-my-service-ext-5s55db85-fw
service.kubernetes.io/healthcheck: k8s2-qvveq1d8-default-my-service-ext-5s55db85
service.kubernetes.io/tcp-forwarding-rule: a808124abf8ce406ca51ab3d4e7d0b7d
Selector: app=my-app
Type: LoadBalancer
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.18.102.23
IPs: 10.18.102.23
LoadBalancer Ingress: 35.184.160.229
Port: tcp-port 8080/TCP
TargetPort: 8080/TCP
NodePort: tcp-port 31864/TCP
Endpoints: 10.20.1.28:8080,10.20.1.29:8080
Session Affinity: None
External Traffic Policy: Local
HealthCheck NodePort: 30394
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ADD 4m55s loadbalancer-controller default/my-service-ext
There are several fields that indicate that a backend service-based external passthrough Network Load Balancer and its Google Cloud resources were successfully created:
Events
field. This field is empty when the LoadBalancer Service and its resources were created successfully. If an error has occurred, it is listed here.List of Annotations
enabled: GKE adds the following list of read-only annotations to the Service manifest. Each annotation whose name begins with service.kubernetes.io/
is used to indicate the name of a Google Cloud resource created as part of or to support the load balancer.
networking.gke.io/weighted-load-balancing: pods-per-node
annotation indicates that weighted load balancing was applied and the load balancer distributes traffic to backend Pods based on the number of Pods running on each node.service.kubernetes.io/backend-service
annotation indicates the name of the load balancer's backend service.service.kubernetes.io/healthcheck
annotation indicates the name of the load balancer health check used by the backend service.service.kubernetes.io/tcp-forwarding-rule
or service.kubernetes.io/udp-forwarding-rule
annotation indicates the name of the load balancer's forwarding rule.service.kubernetes.io/firewall-rule
annotation indicates the name of the firewall rule created to permit traffic to the cluster nodes. Source ranges for this firewall rule are customizable using spec.loadBalancerSourceRanges[]
. For additional detail about firewall rules for LoadBalancer Services, see Firewall rules and source IP address allowlist.service.kubernetes.io/firewall-rule-for-hc
annotation indicates the name of the firewall rule required for load balancer health checks.The cloud.google.com/neg-status
annotation indicates both the NEGs that are used by the load balancer and their zones. This annotation is only present when both of the following are true:
cloud.google.com/l4-rbs: "enabled"
annotation was present in the Service manifest when it was applied to the cluster.Verify that load balancer resources and firewall rules have been created for the external LoadBalancer Service:
To see the forwarding rule, run the following command:
gcloud compute forwarding-rules describe FWD_RULE_NAME \
--region=REGION_NAME
Replace the following:
FWD_RULE_NAME
: the forwarding rule name provided by either the service.kubernetes.io/tcp-forwarding-rule
or service.kubernetes.io/udp-forwarding-rule
read-only annotations. To check these annotations, run kubectl describe svc SERVICE_NAME
.REGION_NAME
: the Google Cloud region containing the cluster. For zonal clusters, the region contains the zone used by the cluster.To see the backend service, run the following command:
gcloud compute backend-services describe BACKEND_SERVICE_NAME \
--region=REGION_NAME
Replace the following:
BACKEND_SERVICE_NAME
: the name of the backend service provided by the service.kubernetes.io/backend-service
read-only annotation. To check this read-only annotation, run kubectl describe svc SERVICE_NAME
.REGION_NAME
: the Google Cloud region containing the cluster. For zonal clusters, the region contains the zone used by the cluster.To see the load balancer health check, run the following command:
gcloud compute health-checks describe HEALTH_CHECK_NAME \
--region=REGION_NAME
Replace the following:
HEALTH_CHECK_NAME
: the load balancer's health check name. The name of the health check is provided by the service.kubernetes.io/healthcheck
read-only annotation. To check this read-only annotation, run kubectl describe svc SERVICE_NAME
.REGION_NAME
: the Google Cloud region containing the cluster. For zonal clusters, the region contains the zone used by the cluster.To see the firewall rules, run the following commands:
gcloud compute firewall-rules describe FIREWALL_RULE_NAME \
gcloud compute firewall-rules describe HEALTH_CHECK_FIREWALL_RULE_NAME
Replace the following:
FIREWALL_RULE_NAME
: the name of the firewall rule that permits traffic to the load balancer. The name of this firewall rule is provided by the service.kubernetes.io/firewall-rule
read-only annotation. To check this read-only annotation, run kubectl describe svc SERVICE_NAME
.HEALTH_CHECK_FIREWALL_RULE_NAME
: the name of the firewall rule that permits health checks of the load balancer's backends (the cluster's nodes). The name of this firewall rule is provided by the service.kubernetes.io/firewall-rule-for-hc
read-only annotation. To check this read-only annotation, run kubectl describe svc SERVICE_NAME
.To see the load balancer NEGs, run the following command:
gcloud compute network-endpoint-groups describe NEG_NAME \
--zone=ZONE_NAME
Replace the following:
NEG_NAME
: the load balancer's NEG name. The name of the NEG is provided by the cloud.google.com/neg-status
read-only annotation. To check this read-only annotation, run the kubectl describe svc SERVICE_NAME
command. The annotation contains structured data with information about the NEG names and zones that are used by the load balancer. For zonal clusters, this annotation contains information about one NEG. For regional clusters, this annotation contains information about a NEG in each zone that the cluster is located in.ZONE_NAME
: the Google Cloud zone that contains the NEG.To delete the sample store-v1-lb-svc
external LoadBalancer Service, use the following command:
kubectl delete service store-v1-lb-svc
GKE automatically removes all load balancer resources that it created for the external LoadBalancer Service.
Migrate toGCE_VM_IP
NEG backends
External LoadBalancer Services with the cloud.google.com/l4-rbs: "enabled"
annotation create backend service-based external passthrough Network Load Balancers that use either GCE_VM_IP
network endpoint group or instance group backends backends, depending on the GKE version of the cluster:
If the Service manifiest was applied to a cluster running GKE version 1.32.2-gke.1652000 or later, the resulting external passthrough Network Load Balancer uses GCE_VM_IP
network endpoint group (NEG) backends.
If the Service manifiest was applied to a cluster running an earlier GKE version, the resulting external passthrough Network Load Balancer uses unmanaged instance group backends.
For more information, see Node grouping in About LoadBalancer Services.
You can create a new external LoadBalancer Service powered by a backend service-based external passthrough Network Load Balancer that uses GCE_VM_IP
NEG backends if your existing Service uses one of the following load balancers:
GCE_VM_IP
NEG backends requires a few minutes of downtime to replace the Service.
To switch to a backend service-based external passthrough Network Load Balancer using GCE_VM_IP
NEG backends:
If you haven't already, upgrade your cluster to GKE version 1.32.2-gke.1652000 or later.
Identify the external LoadBalancer Service that you want to switch to a backend service-based external passthrough Network Load Balancer using GCE_VM_IP
NEG backends. Describe the Service using the following command:
kubectl describe svc SERVICE_NAME -n SERVICE_NAMESPACE
Replace the following:
SERVICE_NAME
: the name of the existing external LoadBalancer Service.
SERVICE_NAMESPACE
: the namespace of the existing external LoadBalancer Service.
In the command output, note the external IPv4 address used by the existing load balancer in the EXTERNAL-IP
column.
Retrieve the Service manifest for the existing LoadBalancer Service:
It's best if you have the original Service manifest that you applied to the cluster in the past. For example, you might have this in a source control repository.
If you don't have the original Service manifest:
Run the following command to get a YAML copy of the Service manifest that represents the load balancer current implementation:
kubectl get svc SERVICE_NAME -n SERVICE_NAMESPACE -o yaml
Copy the manifest YAML to a text editor. Remove the status
attribute and the following metadata
attributes:
kubectl.kubernetes.io/last-applied-configuration
annotationservice.kubernetes.io
creationTimestamp
finalizers
resourceVersion
uid
Ensure that the manifest includes the cloud.google.com/l4-rbs: "enabled"
annotation. If you're migrating from target pool-based external passthrough Network Load Balancer the annotation needs to be added.
Note the local path containing the Service manifest file. The rest of this procedure refers to the path as MANIFEST_FILE_PATH.
Configure a static external IPv4 address resource to hold the external IPv4 address used by the existing load balancer:
gcloud compute addresses create IP_ADDRESS_NAME --region=CLUSTER_REGION --addresses LB_EXTERNAL_IP
Replace the following:
IP_ADDRESS_NAME
: the name of the static external IP address. The name must adhere to name conventions for Compute Engine resources.
CLUSTER_REGION
: the region that contains the cluster. For zonal clusters, this is the region that contains the cluster's zone.
LB_EXTERNAL_IP
: the external IPv4 address used by the current load balancer, determined in the second step of this procedure.
Verify that the static external IPv4 address resource was created:
gcloud compute addresses describe IP_ADDRESS_NAME --region=CLUSTER_REGION
Replace the variables as inidicated in the previous step.
Delete the existing Service:
kubectl delete svc SERVICE_NAME -n SERVICE_NAMESPACE
Add the following annotation to the MANIFEST_FILE_PATH
Service manifest file:
networking.gke.io/load-balancer-ip-addresses: IP_ADDRESS_NAME
For more information about this annotation, see Static IP addresses in LoadBalancer Service parameters.
Apply the updated Service manifest to the cluster:
kubectl apply -f MANIFEST_FILE_PATH
(Optional) Release the static IPv4 address resource.
gcloud compute addresses delete IP_ADDRESS_NAME --region=CLUSTER_REGION
This section describes an issue that you might encounter when you configure weighted load balancing.
Wrong external traffic policy for weighted load balancingIf you don't set externalTrafficPolicy: Local
when you enable weighted load balancing, you might get a warning event when you describe the Service using the following command:
kubectl describe svc store-v1-lb-svc`
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning UnsupportedConfiguration 4m55s loadbalancer-controller Weighted load balancing by pods-per-node has no effect with External Traffic Policy: Cluster.
To effectively enable weighted load balancing, you must set externalTrafficPolicy: Local
.
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4