This page illustrates how to deploy a regional internal Application Load Balancer to load balance traffic to network endpoints that are on-premises or in other public clouds and are reachable by using hybrid connectivity.
If you haven't already done so, review the Hybrid connectivity NEGs overview to understand the network requirements to set up hybrid load balancing.
Setup overviewThe example on this page sets up the following deployment:
Regional internal Application Load Balancer example for hybrid connectivity (click to enlarge).You must configure hybrid connectivity before setting up a hybrid load balancing deployment. This page does not include the hybrid connectivity setup.
Depending on your choice of hybrid connectivity product (either Cloud VPN or Cloud Interconnect (Dedicated or Partner)), use the relevant product documentation.
PermissionsTo set up hybrid load balancing, you must have the following permissions:
On Google Cloud
roles/compute.loadBalancerAdmin
) contains the permissions required to perform the tasks described in this guide.On your on-premises environment or other non-Google Cloud cloud environment
IP:Port
combination. For more information, contact your environment's network administrator.Additionally, to complete the instructions on this page, you need to create a hybrid connectivity NEG, a load balancer, and zonal NEGs (and their endpoints) to serve as Google Cloud-based backends for the load balancer.
You should be either a project Owner or Editor, or you should have the following Compute Engine IAM roles.
Establish hybrid connectivityYour Google Cloud and on-premises environment or other cloud environments must be connected through hybrid connectivity by using either Cloud Interconnect VLAN attachments or Cloud VPN tunnels with Cloud Router or Router appliance VMs. We recommend that you use a high availability connection.
A Cloud Router enabled with global dynamic routing learns about the specific endpoint through Border Gateway Protocol (BGP) and programs it into your Google Cloud VPC network. Regional dynamic routing is not supported. Static routes are also not supported.
You can use either the same network or a different VPC network within the same project to configure both hybrid networking (Cloud Interconnect or Cloud VPN or a Router appliance VM) and the load balancer. Note the following:
If you use different VPC networks, the two networks must be connected using either VPC Network Peering or they must be VPC spokes on the same Network Connectivity Center hub.
If you use the same VPC network, ensure that your VPC network's subnet CIDR ranges don't conflict with your remote CIDR ranges. When IP addresses overlap, subnet routes are prioritized over remote connectivity.
For instructions, see the following documentation:
Important: Don't proceed with the instructions on this page until you set up hybrid connectivity between your environments. Set up your environment that is outside Google CloudPerform the following steps to set up your on-premises environment or other cloud environment for hybrid load balancing:
IP:Port
).After you set up hybrid connectivity, you configure one or more network endpoints within your on-premises environment or other cloud environments that are reachable through Cloud Interconnect or Cloud VPN or Router appliance by using an IP:port
combination. This IP:port
combination is configured as one or more endpoints for the hybrid connectivity NEG that is created in Google Cloud later on in this process.
If there are multiple paths to the IP endpoint, routing follows the behavior described in the Cloud Router overview.
Set up firewall rulesThe following firewall rules must be created on your on-premises environment or other cloud environment:
Allowing traffic from Google's health check probe ranges isn't required for hybrid NEGs. However, if you're using a combination of hybrid and zonal NEGs in a single backend service, you need to allow traffic from the Google health check probe ranges for the zonal NEGs.
Configure Cloud Router to advertise the following custom IP ranges to your on-premises environment or other cloud environment:
For the following steps, make sure you use the same VPC network (called NETWORK in this procedure) that was used to configure hybrid connectivity between the environments.
Additionally, make sure the region used (called REGION in this procedure) is the same as that used to create the Cloud VPN tunnel or Cloud Interconnect VLAN attachment.
Configure the proxy-only subnetThis proxy-only subnet is used for all regional Envoy-based load balancers in the REGION region.
ConsoleCreate the proxy-only subnet with the gcloud compute networks subnets create
command.
gcloud compute networks subnets create PROXY_ONLY_SUBNET_NAME \ --purpose=REGIONAL_MANAGED_PROXY \ --role=ACTIVE \ --region=REGION \ --network=NETWORK \ --range=PROXY_ONLY_SUBNET_RANGEConfigure the load balancer subnet
This subnet is used to create the load balancer's zonal NEG backends, the frontend, and the internal IP address.
Cloud consoleCreate a subnet in the network that was used to configure hybrid connectivity between the environments.
gcloud compute networks subnets create LB_SUBNET_NAME \ --network=NETWORK \ --range=LB_SUBNET_RANGE \ --region=REGION
By default, one IP address is used for each forwarding rule. You can reserve a shared IP address, which lets you use the same IP address with multiple forwarding rules. However, if you want to publish the load balancer by using Private Service Connect, don't use a shared IP address for the forwarding rule.
ConsoleYou can reserve a standalone internal IP address using the Google Cloud console.
Using the gcloud CLI, run the compute addresses create
command:
gcloud compute addresses create LB_IP_ADDRESS \ --region=REGION \ --subnet=LB_SUBNET_NAME \
Use the compute addresses describe
command to view the allocated IP address:
gcloud compute addresses describe LB_IP_ADDRESS \ --region=REGION
If you want to use the same IP address with multiple forwarding rules, specify --purpose=SHARED_LOADBALANCER_VIP
.
In this example, you create the following firewall rules for the zonal NEG backends on Google Cloud:
fw-allow-health-check
: An ingress rule, applicable to the instances being load balanced, that allows traffic from Google Cloud health checking systems (130.211.0.0/22
and 35.191.0.0/16
). This example uses the target tag allow-health-check
to identify the backend VMs to which it should apply. Allowing traffic from Google's health check probe ranges isn't required for hybrid NEGs. However, if you're using a combination of hybrid and zonal NEGs in a single backend service, you need to allow traffic from the Google health check probe ranges for the zonal NEGs.fw-allow-ssh
: An ingress rule that allows incoming SSH connectivity on TCP port 22 from any address. You can choose a more restrictive source IP range for this rule; for example, you can specify just the IP ranges of the systems from which you will initiate SSH sessions. This example uses the target tag allow-ssh
to identify the VMs to which it should apply.fw-allow-proxy-only-subnet
: An ingress rule that allows connections from the proxy-only subnet to reach the backends.
fw-allow-health-check
.allow-health-check
.130.211.0.0/22
and 35.191.0.0/16
.80
for the port number.fw-allow-ssh
1000
allow-ssh
0.0.0.0/0
22
for the port number.fw-allow-proxy-only-subnet
1000
allow-proxy-only-subnet
80
for the port number.Create the fw-allow-health-check-and-proxy
rule to allow the Google Cloud health checks to reach the backend instances on TCP port 80
:
gcloud compute firewall-rules create fw-allow-health-check \ --network=NETWORK \ --action=allow \ --direction=ingress \ --target-tags=allow-health-check \ --source-ranges=130.211.0.0/22,35.191.0.0/16 \ --rules=tcp:80
Create the fw-allow-ssh
firewall rule to allow SSH connectivity to VMs with the network tag allow-ssh
. When you omit source-ranges
, Google Cloud interprets the rule to mean any source.
gcloud compute firewall-rules create fw-allow-ssh \ --network=NETWORK \ --action=allow \ --direction=ingress \ --target-tags=allow-ssh \ --rules=tcp:22
Create an ingress allow firewall rule for the proxy-only subnet to allow the load balancer to communicate with backend instances on TCP port 80
:
gcloud compute firewall-rules create fw-allow-proxy-only-subnet \ --network=NETWORK \ --action=allow \ --direction=ingress \ --target-tags=allow-proxy-only-subnet \ --source-ranges=PROXY_ONLY_SUBNET_RANGE \ --rules=tcp:80
For Google Cloud-based backends, we recommend you configure multiple zonal NEGs in the same region where you configured hybrid connectivity.
For this example, we set up a zonal NEG (with GCE_VM_IP_PORT
type endpoints) in the REGION region. First create the VMs in the GCP_NEG_ZONE zone. Then create a zonal NEG in the same GCP_NEG_ZONE and add the VMs' network endpoints to the NEG.
vm-a1
.Click Advanced options and make the following changes:
allow-ssh
,allow-health-check
, and allow-proxy-only-subnet
.Click Management. In the Startup script field, copy and paste the following script contents. The script contents are identical for all four VMs:
#! /bin/bash apt-get update apt-get install apache2 -y a2ensite default-ssl a2enmod ssl vm_hostname="$(curl -H "Metadata-Flavor:Google" \ http://metadata.google.internal/computeMetadata/v1/instance/name)" echo "Page served from: $vm_hostname" | \ tee /var/www/html/index.html systemctl restart apache2
Click Create.
Repeat the following steps to create a second VM, using the following name and zone combination:
vm-a2
, zone: GCP_NEG_ZONECreate the VMs by running the following command two times, using these combinations for the name of the VM and its zone. The script contents are identical for both VMs.
vm-a1
and any GCP_NEG_ZONE zone of your choiceVM_NAME of vm-a2
and the same GCP_NEG_ZONE zone
gcloud compute instances create VM_NAME \ --zone=GCP_NEG_ZONE \ --image-family=debian-12 \ --image-project=debian-cloud \ --tags=allow-ssh,allow-health-check,allow-proxy-only-subnet \ --subnet=LB_SUBNET_NAME \ --metadata=startup-script='#! /bin/bash apt-get update apt-get install apache2 -y a2ensite default-ssl a2enmod ssl vm_hostname="$(curl -H "Metadata-Flavor:Google" \ http://metadata.google.internal/computeMetadata/v1/instance/name)" echo "Page served from: $vm_hostname" | \ tee /var/www/html/index.html systemctl restart apache2'
To create a zonal network endpoint group:
80
.Add endpoints to the zonal NEG:
80
for all endpoints in the network endpoint group. This is sufficient for our example because the Apache server is serving requests at port 80
.Create a zonal NEG (with GCE_VM_IP_PORT
endpoints) using the gcloud compute network-endpoint-groups create
command:
gcloud compute network-endpoint-groups create GCP_NEG_NAME \ --network-endpoint-type=GCE_VM_IP_PORT \ --zone=GCP_NEG_ZONE \ --network=NETWORK \ --subnet=LB_SUBNET_NAME
You can either specify a --default-port
while creating the NEG, or specify a port number for each endpoint as shown in the next step.
Add endpoints to GCP_NEG_NAME.
gcloud compute network-endpoint-groups update GCP_NEG_NAME \ --zone=GCP_NEG_ZONE \ --add-endpoint='instance=vm-a1,port=80' \ --add-endpoint='instance=vm-a2,port=80'
When creating the NEG, use a ZONE that minimizes the geographic distance between Google Cloud and your on-premises or other cloud environment. For example, if you are hosting a service in an on-premises environment in Frankfurt, Germany, you can specify the europe-west3-a
Google Cloud zone when you create the NEG.
Moreover, if you're using Cloud Interconnect, the ZONE used to create the NEG should be in the same region where the Cloud Interconnect attachment was configured.
For the available regions and zones, see the Compute Engine documentation: Available regions and zones.
ConsoleTo create a hybrid connectivity network endpoint group:
Add endpoints to the hybrid connectivity NEG:
Create a hybrid connectivity NEG using the gcloud compute network-endpoint-groups create
command.
gcloud compute network-endpoint-groups create ON_PREM_NEG_NAME \ --network-endpoint-type=NON_GCP_PRIVATE_IP_PORT \ --zone=ON_PREM_NEG_ZONE \ --network=NETWORK
Add the on-premises backend VM endpoint to ON_PREM_NEG_NAME:
gcloud compute network-endpoint-groups update ON_PREM_NEG_NAME \ --zone=ON_PREM_NEG_ZONE \ --add-endpoint="ip=ON_PREM_IP_ADDRESS_1,port=PORT_1" \ --add-endpoint="ip=ON_PREM_IP_ADDRESS_2,port=PORT_2"
You can use this command to add the network endpoints you previously configured on-premises or in your cloud environment. Repeat --add-endpoint
as many times as needed.
You can repeat these steps to create multiple hybrid NEGs if needed.
Configure the load balancer Console Note: You cannot use the Google Cloud console to create a load balancer that has mixed zonal and hybrid connectivity NEG backends in a single backend service. Use eithergcloud
or the REST API instead. gcloud
gcloud compute health-checks create http HTTP_HEALTH_CHECK_NAME \ --region=REGION \ --use-serving-portHealth check probes for hybrid NEG backends originate from Envoy proxies in the proxy-only subnet, whereas probes for zonal NEG backends originate from [Google's central probe IP ranges](/load-balancing/docs/health-check-concepts#ip-ranges).
gcloud compute backend-services create BACKEND_SERVICE \ --load-balancing-scheme=INTERNAL_MANAGED \ --protocol=HTTP \ --health-checks=HTTP_HEALTH_CHECK_NAME \ --health-checks-region=REGION \ --region=REGION
gcloud compute backend-services add-backend BACKEND_SERVICE \ --region=REGION \ --balancing-mode=RATE \ --max-rate-per-endpoint=MAX_REQUEST_RATE_PER_ENDPOINT \ --network-endpoint-group=GCP_NEG_NAME \ --network-endpoint-group-zone=GCP_NEG_ZONEFor details about configuring the balancing mode, see the gcloud CLI documentation for the
--max-rate-per-endpoint
parameter.gcloud compute backend-services add-backend BACKEND_SERVICE \ --region=REGION \ --balancing-mode=RATE \ --max-rate-per-endpoint=MAX_REQUEST_RATE_PER_ENDPOINT \ --network-endpoint-group=ON_PREM_NEG_NAME \ --network-endpoint-group-zone=ON_PREM_NEG_ZONEFor details about configuring the balancing mode, see the gcloud CLI documentation for the
--max-rate-per-endpoint
parameter.gcloud compute url-maps create URL_MAP_NAME \ --default-service BACKEND_SERVICE \ --region=REGION
You can create either Compute Engine or Certificate Manager certificates. Use any of the following methods to create certificates using Certificate Manager:
Regional Google-managed certificates. Certificate maps aren't supported.
The following types of regional Google-managed certificates are supported by Certificate Manager:
After you create certificates, attach the certificate directly to the target proxy.
To create a Compute Engine self-managed SSL certificate resource:gcloud compute ssl-certificates create SSL_CERTIFICATE_NAME \ --certificate CRT_FILE_PATH \ --private-key KEY_FILE_PATH
For an HTTP load balancer, create an HTTP target proxy:
gcloud compute target-http-proxies create TARGET_HTTP_PROXY_NAME \ --url-map=URL_MAP_NAME \ --url-map-region=REGION \ --region=REGIONFor an HTTPS load balancer, create an HTTPS target proxy. The proxy is the portion of the load balancer that holds the SSL certificate for HTTPS Load Balancing, so you also load your certificate in this step.
gcloud compute target-https-proxies create TARGET_HTTPS_PROXY_NAME \ --ssl-certificates=SSL_CERTIFICATE_NAME \ --url-map=URL_MAP_NAME \ --url-map-region=REGION \ --region=REGION
For an HTTP load balancer:
gcloud compute forwarding-rules create HTTP_FORWARDING_RULE_NAME \ --load-balancing-scheme=INTERNAL_MANAGED \ --network=NETWORK \ --subnet=LB_SUBNET_NAME \ --address=LB_IP_ADDRESS \ --ports=80 \ --region=REGION \ --target-http-proxy=TARGET_HTTP_PROXY_NAME \ --target-http-proxy-region=REGIONFor an HTTPS load balancer:
gcloud compute forwarding-rules create HTTPS_FORWARDING_RULE_NAME \ --load-balancing-scheme=INTERNAL_MANAGED \ --network=NETWORK \ --subnet=LB_SUBNET_NAME \ --address=LB_IP_ADDRESS \ --ports=443 \ --region=REGION \ --target-https-proxy=TARGET_HTTPS_PROXY_NAME \ --target-https-proxy-region=REGION
After the load balancer is created, note the IP address that is associated with the load balancer—for example, 30.90.80.100
. To point your domain to your load balancer, create an A
record by using your domain registration service. If you added multiple domains to your SSL certificate, you must add an A
record for each one, all pointing to the load balancer's IP address. For example, to create A
records for www.example.com
and example.com
, use the following:
NAME TYPE DATA www A 30.90.80.100 @ A 30.90.80.100
If you use Cloud DNS as your DNS provider, see Add, modify, and delete records.
Test the load balancerTo test the load balancer, create a client VM in the same region as the load balancer. Then send traffic from the client to the load balancer.
Create a client VMThis example creates a client VM (vm-client
) in the same region as the backend NEGs. The client is used to validate the load balancer's configuration and demonstrate expected behavior as described in the testing section.
vm-client
.allow-ssh
to Network tags.The client VM can be in any zone in the same region as the load balancer, and it can use any subnet in that region. In this example, the client is in the CLIENT_VM_ZONE zone, and it uses the same subnet as the backend VMs.
gcloud compute instances create vm-client \ --zone=CLIENT_VM_ZONE \ --image-family=debian-12 \ --image-project=debian-cloud \ --tags=allow-ssh \ --subnet=LB_SUBNET_NAMESend traffic to the load balancer Note: It might take a few minutes for the load balancer configuration to propagate globally after you first deploy it.
Now that you have configured your load balancer, you can start sending traffic to the load balancer's IP address.
Use SSH to connect to the client instance.
gcloud compute ssh client-vm \ --zone=CLIENT_VM_ZONE
Get the load balancer's IP address. Use the compute addresses describe
command to view the allocated IP address:
gcloud compute addresses describe l7-ilb-ip-address \ --region=us-west1
Verify that the load balancer is serving backend hostnames as expected. Replace IP_ADDRESS with the load balancer's IP address.
For HTTP testing, run:
curl IP_ADDRESS
For HTTPS testing, run:
curl -k -s 'https://DOMAIN_NAME:443' --connect-to DOMAIN_NAME:443:IP_ADDRESS:443
Replace DOMAIN_NAME with your application domain name, for example, test.example.com
.
The -k
flag causes curl to skip certificate validation.
Testing the non-Google Cloud endpoints depends on the service you have exposed through the hybrid NEG endpoint.
Additional configuration optionsThis section expands on the configuration example to provide alternative and additional configuration options. All of the tasks are optional. You can perform them in any order.
Update client HTTP keepalive timeoutThe load balancer created in the previous steps has been configured with a default value for the
client HTTP keepalive timeout.
To update the client HTTP keepalive timeout, use the following instructions.
ConsoleIn the Google Cloud console, go to the Load balancing page.
For an HTTP load balancer, update the target HTTP proxy by using the gcloud compute target-http-proxies update
command.
gcloud compute target-http-proxies update TARGET_HTTP_PROXY_NAME \ --http-keep-alive-timeout-sec=HTTP_KEEP_ALIVE_TIMEOUT_SEC \ --region=REGION
For an HTTPS load balancer, update the target HTTPS proxy by using the gcloud compute target-https-proxies update
command.
gcloud compute target-https-proxies update TARGET_HTTP_PROXY_NAME \ --http-keep-alive-timeout-sec=HTTP_KEEP_ALIVE_TIMEOUT_SEC \ --region REGION
Replace the following:
TARGET_HTTP_PROXY_NAME
: the name of the target HTTP proxy.TARGET_HTTPS_PROXY_NAME
: the name of the target HTTPS proxy.HTTP_KEEP_ALIVE_TIMEOUT_SEC
: the HTTP keepalive timeout value from 5 to 600 seconds.RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4