This page shows how to deploy a cross-region internal proxy Network Load Balancer to load balance traffic to network endpoints that are on-premises or in other public clouds and that are reachable by using hybrid connectivity.
If you haven't already done so, review the Hybrid connectivity NEGs overview to understand the network requirements to set up hybrid load balancing.
Setup overviewThe example sets up a cross-region internal proxy Network Load Balancer for mixed zonal and hybrid connectivity NEG backends, as shown in the following figure:
Cross-region internal proxy Network Load Balancer example for mixed zonal and hybrid connectivity NEG backends (click to enlarge).You must configure hybrid connectivity before setting up a hybrid load balancing deployment. Depending on your choice of hybrid connectivity product, use either Cloud VPN or Cloud Interconnect (Dedicated or Partner).
PermissionsTo set up hybrid load balancing, you must have the following permissions:
On Google Cloud
roles/compute.loadBalancerAdmin
) contains the permissions required to perform the tasks described in this guide.On your on-premises environment or other non-Google Cloud cloud environment
IP:Port
combination. For more information, contact your environment's network administrator.Additionally, to complete the instructions on this page, you need to create a hybrid connectivity NEG, a load balancer, and zonal NEGs (and their endpoints) to serve as Google Cloud-based backends for the load balancer.
You should be either a project Owner or Editor, or you should have the following Compute Engine IAM roles.
Establish hybrid connectivityYour Google Cloud and on-premises environment or other cloud environments must be connected through hybrid connectivity by using either Cloud Interconnect VLAN attachments or Cloud VPN tunnels with Cloud Router or Router appliance VMs. We recommend that you use a high availability connection.
A Cloud Router enabled with global dynamic routing learns about the specific endpoint through Border Gateway Protocol (BGP) and programs it into your Google Cloud VPC network. Regional dynamic routing is not supported. Static routes are also not supported.
You can use either the same network or a different VPC network within the same project to configure both hybrid networking (Cloud Interconnect or Cloud VPN or a Router appliance VM) and the load balancer. Note the following:
If you use different VPC networks, the two networks must be connected using either VPC Network Peering or they must be VPC spokes on the same Network Connectivity Center hub.
If you use the same VPC network, ensure that your VPC network's subnet CIDR ranges don't conflict with your remote CIDR ranges. When IP addresses overlap, subnet routes are prioritized over remote connectivity.
For instructions, see the following documentation:
Important: Don't proceed with the instructions on this page until you set up hybrid connectivity between your environments. Set up your environment that is outside Google CloudPerform the following steps to set up your on-premises environment or other cloud environment for hybrid load balancing:
IP:Port
).After you set up hybrid connectivity, you configure one or more network endpoints within your on-premises environment or other cloud environments that are reachable through Cloud Interconnect or Cloud VPN or Router appliance by using an IP:port
combination. This IP:port
combination is configured as one or more endpoints for the hybrid connectivity NEG that is created in Google Cloud later on in this process.
If there are multiple paths to the IP endpoint, routing follows the behavior described in the Cloud Router overview.
Set up firewall rulesThe following firewall rules must be created on your on-premises environment or other cloud environment:
Allowing traffic from Google's health check probe ranges isn't required for hybrid NEGs. However, if you're using a combination of hybrid and zonal NEGs in a single backend service, you need to allow traffic from the Google health check probe ranges for the zonal NEGs.
Configure Cloud Router to advertise the following custom IP ranges to your on-premises environment or other cloud environment:
For the following steps, make sure you use the same VPC network (called NETWORK
in this procedure) that was used to configure hybrid connectivity between the environments.
Additionally, make sure the regions used (called REGION_A
and REGION_B
in this procedure) are the same as those used to create the Cloud VPN tunnel or Cloud Interconnect VLAN attachments.
Use this subnet to create the load balancer's zonal NEG backends:
ConsoleIn the Google Cloud console, go to the VPC networks page.
Go to the network that was used to configure hybrid connectivity between the environments.
In the Subnets section:
Click Create.
To add more subnets in different regions, click Add subnet and repeat the previous steps for REGION_B
Create subnets in the network that was used to configure hybrid connectivity between the environments.
gcloud compute networks subnets create SUBNET_A \ --network=NETWORK \ --range=LB_SUBNET_RANGE1 \ --region=REGION_A
gcloud compute networks subnets create SUBNET_B \ --network=NETWORK \ --range=LB_SUBNET_RANGE2 \ --region=REGION_B
Make a POST
request to the subnetworks.insert
method. Replace PROJECT_ID
with your project ID.
POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/REGION_A/subnetworks { "name": "SUBNET_A", "network": "projects/PROJECT_ID/global/networks/NETWORK", "ipCidrRange": "LB_SUBNET_RANGE1", "region": "projects/PROJECT_ID/regions/REGION_A", }
Make a POST
request to the subnetworks.insert
method. Replace PROJECT_ID
with your project ID.
POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/REGION_B/subnetworks { "name": "SUBNET_B", "network": "projects/PROJECT_ID/global/networks/NETWORK", "ipCidrRange": "LB_SUBNET_RANGE2", "region": "projects/PROJECT_ID/regions/REGION_B", }
Replace the following:
SUBNET_A
and SUBNET_B
: the name of the subnetsLB_SUBNET_RANGE1
and LB_SUBNET_RANGE2
: the IP address range for the subnetsREGION_A
and REGION_B
: the regions where you have configured the load balancerA proxy-only subnet provides a set of IP addresses that Google uses to run Envoy proxies on your behalf. The proxies terminate connections from the client and create new connections to the backends.
This proxy-only subnet is used by all Envoy-based regional load balancers in the same region of the VPC network. There can only be one active proxy-only subnet for a given purpose, per region, per network.
Important: Don't try to assign addresses from the proxy-only subnet to your load balancer's forwarding rule or backends. You assign the forwarding rule's IP address and the backend instance IP addresses from a different subnet range (or ranges), not this one. Google Cloud reserves this subnet range for Google Cloud-managed proxies. ConsoleIf you're using the Google Cloud console, you can wait and create the proxy-only subnet later on the Load balancing page.
If you want to create the proxy-only subnet now, use the following steps:
In the Google Cloud console, go to the VPC networks page.
10.129.0.0/23
.Create the proxy-only subnet in REGION_B
10.130.0.0/23
.Create the proxy-only subnets with the gcloud compute networks subnets create
command.
gcloud compute networks subnets create PROXY_SN_A \ --purpose=GLOBAL_MANAGED_PROXY \ --role=ACTIVE \ --region=REGION_A \ --network=NETWORK \ --range=PROXY_ONLY_SUBNET_RANGE1
gcloud compute networks subnets create PROXY_SN_B \ --purpose=GLOBAL_MANAGED_PROXY \ --role=ACTIVE \ --region=REGION_B \ --network=NETWORK \ --range=PROXY_ONLY_SUBNET_RANGE2
Replace the following:
PROXY_SN_A
and PROXY_SN_B
: the name of the proxy-only subnetsPROXY_ONLY_SUBNET_RANGE1
and PROXY_ONLY_SUBNET_RANGE2
: the IP address range for the proxy-only subnetsREGION_A
and REGION_B
: the regions where you have configured the load balancerCreate the proxy-only subnets with the subnetworks.insert
method, replacing PROJECT_ID
with your project ID.
POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/REGION_A/subnetworks { "name": "PROXY_SN_A", "ipCidrRange": "PROXY_ONLY_SUBNET_RANGE1", "network": "projects/PROJECT_ID/global/networks/NETWORK", "region": "projects/PROJECT_ID/regions/REGION_A", "purpose": "GLOBAL_MANAGED_PROXY", "role": "ACTIVE" }
POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/REGION_B/subnetworks { "name": " PROXY_SN_B", "ipCidrRange": "PROXY_ONLY_SUBNET_RANGE2", "network": "projects/PROJECT_ID/global/networks/NETWORK", "region": "projects/PROJECT_ID/regions/REGION_B", "purpose": "GLOBAL_MANAGED_PROXY", "role": "ACTIVE" }Create firewall rules
In this example, you create the following firewall rules for the zonal NEG backends on Google Cloud:
fw-allow-health-check
: An ingress rule, applicable to the instances being load balanced, that allows traffic from Google Cloud health checking systems (130.211.0.0/22
and 35.191.0.0/16
). This example uses the target tag allow-health-check
to identify the zonal NEGs to which it should apply.fw-allow-ssh
: An ingress rule that allows incoming SSH connectivity on TCP port 22 from any address. You can choose a more restrictive source IP range for this rule; for example, you can specify just the IP ranges of the systems from which you will initiate SSH sessions. This example uses the target tag allow-ssh
to identify the VMs to which it should apply.fw-allow-proxy-only-subnet
: An ingress rule that allows connections from the proxy-only subnet to reach the zonal NEG backends.
In the Google Cloud console, go to the Firewall policies page.
Click Create firewall rule to create the rule to allow traffic from health check probes:
fw-allow-health-check
.allow-health-check
.130.211.0.0/22
and 35.191.0.0/16
.80
for the port number.Click Create firewall rule again to create the rule to allow incoming SSH connections:
fw-allow-ssh
1000
allow-ssh
0.0.0.0/0
22
for the port number.Click Create firewall rule again to create the rule to allow incoming connections from the proxy-only subnet:
fw-allow-proxy-only-subnet
1000
allow-proxy-only-subnet
80
for the port number.Create the fw-allow-health-check-and-proxy
rule to allow the Google Cloud health checks to reach the backend instances on TCP port 80
:
gcloud compute firewall-rules create fw-allow-health-check \ --network=NETWORK \ --action=allow \ --direction=ingress \ --target-tags=allow-health-check \ --source-ranges=130.211.0.0/22,35.191.0.0/16 \ --rules=tcp:80
Create the fw-allow-ssh
firewall rule to allow SSH connectivity to VMs with the network tag allow-ssh
. When you omit source-ranges
, Google Cloud interprets the rule to mean any source.
gcloud compute firewall-rules create fw-allow-ssh \ --network=NETWORK \ --action=allow \ --direction=ingress \ --target-tags=allow-ssh \ --rules=tcp:22
Create an ingress allow firewall rule for the proxy-only subnet to allow the load balancer to communicate with backend instances on TCP port 80
:
gcloud compute firewall-rules create fw-allow-proxy-only-subnet \ --network=NETWORK \ --action=allow \ --direction=ingress \ --target-tags=allow-proxy-only-subnet \ --source-ranges=PROXY_ONLY_SUBNET_RANGE1,PROXY_ONLY_SUBNET_RANGE2 \ --rules=tcp:80
For Google Cloud-based backends, we recommend you configure multiple zonal NEGs in the same region where you configured hybrid connectivity.
For this example, we set up a zonal NEG (with GCE_VM_IP_PORT
type endpoints) in the REGION1
. First create the VMs in the NEG_ZONE1
zone. Then create a zonal NEG in the NEG_ZONE2
and add the VMs' network endpoints to the NEG. To support high availability, we set up a similar zonal NEG in the REGION2 region. If backends in one region happen to be down, traffic fails over to the other region.
In the Google Cloud console, go to the VM instances page.
Repeat steps 3 to 8 for each VM, using the following name and zone combinations.
vm-a1
vm-b1
Click Create instance.
Set the name as indicated in the preceding step.
For the Region, choose as indicated in the earlier step.
For the Zone, choose as indicated in the earlier step.
In the Boot disk section, ensure that Debian GNU/Linux 12 (bookworm) is selected for the boot disk options. Click Choose to change the image if necessary.
In the Advanced options section, expand Networking, and then do the following:
allow-ssh
,allow-health-check
, and allow-proxy-only-subnet
.Expand Management. In the Automation field, copy and paste the following script contents. The script contents are identical for all VMs:
#! /bin/bash apt-get update apt-get install apache2 -y a2ensite default-ssl a2enmod ssl vm_hostname="$(curl -H "Metadata-Flavor:Google" \ http://metadata.google.internal/computeMetadata/v1/instance/name)" echo "Page served from: $vm_hostname" | \ tee /var/www/html/index.html systemctl restart apache2
Click Create.
Create the VMs by running the following command, using these combinations for the name of the VM and its zone. The script contents are identical for both VMs.
VM_NAME
of vm-a1
GCP_NEG_ZONE
as NEG_ZONE1 in the region REGION_A
LB_SUBNET_NAME
as SUBNET_AVM_NAME
of vm-b1
GCP_NEG_ZONE
as NEG_ZONE2 in the region REGION_B
LB_SUBNET_NAME
as SUBNET_Bgcloud compute instances create VM_NAME \ --zone=GCP_NEG_ZONE \ --image-family=debian-12 \ --image-project=debian-cloud \ --tags=allow-ssh,allow-health-check,allow-proxy-only-subnet \ --subnet=LB_SUBNET_NAME \ --metadata=startup-script='#! /bin/bash apt-get update apt-get install apache2 -y a2ensite default-ssl a2enmod ssl vm_hostname="$(curl -H "Metadata-Flavor:Google" \ http://metadata.google.internal/computeMetadata/v1/instance/name)" echo "Page served from: $vm_hostname" | \ tee /var/www/html/index.html systemctl restart apache2'
To create a zonal network endpoint group:
In the Google Cloud console, go to the Network Endpoint Groups page.
Repeat steps 3 to 8 for each zonal NEG, using the following name and zone combinations:
neg-1
REGION_A
neg-2
REGION_B
Click Create network endpoint group.
Set the name as indicated in the preceding step.
Select the Network endpoint group type: Network endpoint group (Zonal).
Select the Network: NETWORK
Select the Subnetwork as indicated in earlier step.
Select the Zone as indicated in earlier step.
Enter the Default port: 80
.
Click Create.
Add endpoints to the zonal NEG:
In the Google Cloud console, go to the Network Endpoint Groups page.
Click the Name of the network endpoint group created in the previous step. You see the Network endpoint group details page.
In the Network endpoints in this group section, click Add network endpoint. You see the Add network endpoint page.
Select a VM instance to add its internal IP addresses as network endpoints. In the Network interface section, the name, zone, and subnet of the VM is displayed.
Enter the IP address of the new network endpoint.
Select the Port type.
80
for all endpoints in the network endpoint group. This is sufficient for our example because the Apache server is serving requests at port 80
.To add more endpoints, click Add network endpoint and repeat the previous steps.
After you add all the endpoints, click Create.
Create zonal NEGs (with GCE_VM_IP_PORT
endpoints) using the name, zone, and subnet combinations. Use the gcloud compute network-endpoint-groups create
command.
neg-1
GCP_NEG_ZONE
: NEG_ZONE1 in the region REGION_A
LB_SUBNET_NAME
: SUBNET_Aneg-2
GCP_NEG_ZONE
: NEG_ZONE2 in the region REGION_B
LB_SUBNET_NAME
: SUBNET_Bgcloud compute network-endpoint-groups create GCP_NEG_NAME \ --network-endpoint-type=GCE_VM_IP_PORT \ --zone=GCP_NEG_ZONE \ --network=NETWORK \ --subnet=LB_SUBNET_NAME
You can either specify a port using the --default-port
option while creating the NEG, or specify a port number for each endpoint as shown in the next step.
Add endpoints to neg1
and neg2
.
gcloud compute network-endpoint-groups update neg1 \ --zone=NEG_ZONE1 \ --add-endpoint='instance=vm-a1,port=80'
gcloud compute network-endpoint-groups update neg2 \ --zone=NEG_ZONE2 \ --add-endpoint='instance=vm-b1,port=80'
When creating the NEG, use a zone that minimizes the geographic distance between Google Cloud and your on-premises or other cloud environment.
And, if you're using Cloud Interconnect, the zone used to create the NEG is in the same region where the Cloud Interconnect attachment was configured.
Hybrid NEGs support only the distributed Envoy health checks.
ConsoleTo create a hybrid connectivity network endpoint group:
In the Google Cloud console, go to the Network Endpoint Groups page.
Click Create network endpoint group.
Repeat steps 4 to 9 for each hybrid NEG, using the following name and zone combinations.
hybrid-1
hybrid-2
Set the name as indicated in the previous step.
Select the Network endpoint group type: Hybrid connectivity network endpoint group (Zonal).
Select the Network: NETWORK
For the Subnet, choose as indicated in the previous step.
For the Zone, choose as indicated in the previous step.
Enter the Default port.
Click Create
Add endpoints to the hybrid connectivity NEG:
In the Google Cloud console, go to the Network Endpoint Groups page.
Click the Name of the network endpoint group created in the previous step. You see the Network endpoint group detail page.
In the Network endpoints in this group section, click Add network endpoint. You see the Add network endpoint page.
Enter the IP address of the new network endpoint.
Select the Port type.
To add more endpoints, click Add network endpoint and repeat the previous steps.
After you add all the non-Google Cloud endpoints, click Create.
Create a hybrid connectivity NEG that uses the following name combinations. Use the gcloud compute network-endpoint-groups create
command.
ON_PREM_NEG_NAME
: hybrid-1
ON_PREM_NEG_ZONE
: ON_PREM_NEG_ZONE1ON_PREM_NEG_NAME
: hybrid-2
GCP_NEG_ZONE
: ON_PREM_NEG_ZONE2gcloud compute network-endpoint-groups create ON_PREM_NEG_NAME \ --network-endpoint-type=NON_GCP_PRIVATE_IP_PORT \ --zone=ON_PREM_NEG_ZONE \ --network=NETWORK
Add the on-premises backend VM endpoint to ON_PREM_NEG_NAME:
gcloud compute network-endpoint-groups update ON_PREM_NEG_NAME \ --zone=ON_PREM_NEG_ZONE \ --add-endpoint="ip=ON_PREM_IP_ADDRESS_1,port=PORT_1" \ --add-endpoint="ip=ON_PREM_IP_ADDRESS_2,port=PORT_2"
You can use this command to add the network endpoints you previously configured on premises or in your cloud environment. Repeat --add-endpoint
as many times as needed.
Define the TCP health check with the gcloud compute health-checks create tcp
command.
gcloud compute health-checks create tcp gil4-basic-check \ --use-serving-port \ --global
Create the backend service and enable logging with the gcloud compute backend-services create
command.
gcloud compute backend-services create BACKEND_SERVICE \ --load-balancing-scheme=INTERNAL_MANAGED \ --protocol=TCP \ --enable-logging \ --logging-sample-rate=1.0 \ --health-checks=gil4-basic-check \ --global-health-checks \ --global
Add backends to the backend service with the gcloud compute backend-services add-backend
command.
gcloud compute backend-services add-backend BACKEND_SERVICE \ --global \ --balancing-mode=CONNECTION \ --max-connections-per-endpoint=MAX_CONNECTIONS \ --network-endpoint-group=neg1 \ --network-endpoint-group-zone=NEG_ZONE1 \ --network-endpoint-group=neg2 \ --network-endpoint-group-zone=NEG_ZONE2
For details about configuring the balancing mode, see the gcloud CLI documentation for the --max-connections-per-endpoint
flag. For MAX_CONNECTIONS, enter the maximum concurrent connections for the backend to handle.
Add the hybrid NEGs as a backend to the backend service.
gcloud compute backend-services add-backend BACKEND_SERVICE \ --global \ --balancing-mode=CONNECTION \ --max-connections-per-endpoint=MAX_CONNECTIONS \ --network-endpoint-group=hybrid1 \ --network-endpoint-group-zone=ON_PREM_NEG_ZONE1 \ --network-endpoint-group=hybrid2 \ --network-endpoint-group-zone=ON_PREM_NEG_ZONE2 \
For details about configuring the balancing mode, see the gcloud CLI documentation for the --max-connections-per-endpoint
parameter. For MAX_CONNECTIONS
, enter the maximum concurrent connections for the backend to handle.
Create the target proxy.
Create the target proxy with the gcloud compute target-tcp-proxies create
command.
gcloud compute target-tcp-proxies create gil4-tcp-proxy \ --backend-service=BACKEND_SERVICE \ --global
Create two forwarding rules, one with a VIP IP_ADDRESS1 in REGION_A and another one with a VIP IP_ADDRESS2 in REGION_B
. For the forwarding rule's IP address, use the LB_SUBNET_RANGE1 or LB_SUBNET_RANGE2 IP address range. If you try to use the proxy-only subnet, forwarding rule creation fails.
For custom networks, you must reference the subnet in the forwarding rule. Note that this is the VM subnet, not the proxy subnet.
Use the gcloud compute forwarding-rules create
command with the correct flags.
gcloud compute forwarding-rules create gil4-forwarding-rule-a \ --load-balancing-scheme=INTERNAL_MANAGED \ --network=NETWORK \ --subnet=SUBNET_A \ --subnet-region=REGION_A \ --address=IP_ADDRESS1 \ --ports=80 \ --target-tcp-proxy=gil4-tcp-proxy \ --global
gcloud compute forwarding-rules create gil4-forwarding-rule-b \ --load-balancing-scheme=INTERNAL_MANAGED \ --network=NETWORK \ --subnet=SUBNET_B \ --subnet-region=REGION_B \ --address=IP_ADDRESS2 \ --ports=80 \ --target-tcp-proxy=gil4-tcp-proxy \ --global
Create client VMs in REGION_A
and REGION_B
and regions:
gcloud compute instances create l4-ilb-client-a \ --image-family=debian-12 \ --image-project=debian-cloud \ --network=NETWORK \ --subnet=SUBNET_A \ --zone=NEG_ZONE1 \ --tags=allow-ssh
gcloud compute instances create l4-ilb-client-b \ --image-family=debian-12 \ --image-project=debian-cloud \ --network=NETWORK \ --subnet=SUBNET_B \ --zone=NEG_ZONE2 \ --tags=allow-ssh
Use SSH to connect to each client instance.
gcloud compute ssh l4-ilb-client-a \ --zone=NEG_ZONE1
gcloud compute ssh l4-ilb-client-b \ --zone=NEG_ZONE2
Verify that the IP address is serving its hostname.
Verify that the client VM can reach both IP addresses. The command should succeed and return the name of the backend VM which served the request:
curl IP_ADDRESS1
curl IP_ADDRESS2
Run 100 curl requests and confirm from the responses that they are load balanced.
Verify that the client VM can reach both IP addresses. The command should succeed and return the name of the backend VM which served the request:
{ RESULTS= for i in {1..100} do RESULTS="$RESULTS:$(curl --silent IP_ADDRESS1)" done echo "***" echo "*** Results of load-balancing to IP_ADDRESS1: " echo "***" echo "$RESULTS" | tr ':' '\n' | grep -Ev "^$" | sort | uniq -c echo }
{ RESULTS= for i in {1..100} do RESULTS="$RESULTS:$(curl --silent IP_ADDRESS2)" done echo "***" echo "*** Results of load-balancing to IP_ADDRESS2: " echo "***" echo "$RESULTS" | tr ':' '\n' | grep -Ev "^$" | sort | uniq -c echo }
Verify failover to backends in the REGION_A
region when backends in the REGION_B
are unhealthy or unreachable. We simulate this by removing all the backends from REGION_B
:
gcloud compute backend-services remove-backend BACKEND_SERVICE \ --balancing-mode=CONNECTION \ --network-endpoint-group=neg2 \ --network-endpoint-group-zone=NEG_ZONE2
Use SSH to connect to the client VM in REGION_B
.
gcloud compute ssh l4-ilb-client-b \ --zone=NEG_ZONE2
Send requests to the load balanced IP address in REGION_B
region. The command output should display responses from backend VMs in REGION_A
.
{ RESULTS= for i in {1..100} do RESULTS="$RESULTS:$(curl --silent IP_ADDRESS2)" done echo "***" echo "*** Results of load-balancing to IP_ADDRESS2: " echo "***" echo "$RESULTS" | tr ':' '\n' | grep -Ev "^$" | sort | uniq -c echo }
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4