This page explains how to configure clusters created in Google Kubernetes Engine (GKE) Standard mode to perform IP masquerade with the ip-masq-agent
. For more information about IP masquerading in GKE Autopilot mode, see Use Egress NAT Policy to configure IP masquerade in Autopilot clusters.
Before you start, make sure that you have performed the following tasks:
gcloud components update
. Note: For existing gcloud CLI installations, make sure to set the compute/region
property. If you use primarily zonal clusters, set the compute/zone
instead. By setting a default location, you can avoid errors in the gcloud CLI like the following: One of [--zone, --region] must be supplied: Please specify location
. You might need to specify the location in certain commands if the location of your cluster differs from the default that you set.ip-masq-agent
status
This section shows you how to:
ip-masq-agent
DaemonSet.ip-masq-agent
ConfigMap resource.ip-masq-agent
DaemonSet
To check if your cluster is running the ip-masq-agent
DaemonSet, use the Google Cloud CLI or the Google Cloud console.
Get the credentials for your cluster:
gcloud container clusters get-credentials CLUSTER_NAME
Replace CLUSTER_NAME
with the name of your cluster.
Search for the ip-masq-agent
in the kube-system
namespace:
kubectl get daemonsets/ip-masq-agent -n kube-system
If the ip-masq-agent
DaemonSet exists, then the output is similar to the following:
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
ip-masq-agent 3 3 3 3 3 <none> 13d
If the ip-masq-agent
DaemonSet does not exist, then the output is similar to the following:
Error from server (NotFound): daemonsets.apps "ip-masq-agent" not found
Check if the ip-masq-agent
DaemonSet is running the latest version:
kubectl get daemonsets/ip-masq-agent -n kube-system -o jsonpath='{.spec.template.spec.containers[].image}'
This command must return the same container image as specified in Deploying the ip-masq-agent DaemonSet.
Go to the Workloads page in the Google Cloud console.
For filter_list Filter, do the following:
ip-masq-agent
.If ip-masq-agent
DaemonSet exists, then you can see the DaemonSet record in the table. If ip-masq-agent
DaemonSet does not exist, then no rows are displayed.
To create the ip-masq-agent
ConfigMap and deploy the ip-masq-agent
DaemonSet, refer to Configuring and deploying the ip-masq-agent
.
ip-masq-agent
ConfigMap
To check if your cluster is running the ip-masq-agent
ConfigMap, use the Google Cloud CLI or the Google Cloud console.
Get the credentials for your cluster:
gcloud container clusters get-credentials CLUSTER_NAME
Replace CLUSTER_NAME
with the name of your cluster.
Describe the ip-masq-agent
ConfigMap in the kube-system
namespace:
kubectl describe configmaps/ip-masq-agent -n kube-system
If the ip-masq-agent
ConfigMap exists, then the output is similar to the following:
Name: ip-masq-agent
Namespace: kube-system
Labels: <none>
Annotations: <none>
Data
====
config:
----
nonMasqueradeCIDRs:
- 198.15.5.92/24
- 10.0.0.0/8
masqLinkLocal: false
resyncInterval: 60s
BinaryData
====
Events: <none>
If the ip-masq-agent
ConfigMap does not exist, then the output is similar to the following:
Error from server (NotFound): configmaps "ip-masq-agent" not found
Go to the Configuration page in the Google Cloud console.
For filter_list Filter, do the following:
ip-masq-agent
.If ip-masq-agent
ConfigMap exists, then you can see the ConfigMap record in the table. If ip-masq-agent
ConfigMap does not exist, then no rows are displayed.
If the cluster already has the ip-masq-agent
ConfigMap, you can configure and deploy it.
ip-masq-agent
This section shows you how to create or edit the ip-masq-agent
ConfigMap and how to deploy or delete the ip-masq-agent
DaemonSet. To determine which tasks you need to perform, you must first determine whether your cluster already has the ip-masq-agent
ConfigMap and ip-masq-agent
DaemonSet.
ip-masq-agent
ConfigMap
The following steps show how to create the ip-masq-agent
ConfigMap. If your cluster already has the ip-masq-agent
ConfigMap, edit an existing ip-masq-agent ConfigMap instead.
Create a configuration file using the following template and save it locally. You can use any name for the local copy of this configuration file.
nonMasqueradeCIDRs:
- CIDR_1
- CIDR_2
masqLinkLocal: false
resyncInterval: SYNC_INTERVALUNIT_OF_TIME
Replace the following:
CIDR_1
and CIDR_2
: the IP address ranges in CIDR format. When packets are sent to these destinations, your cluster does not masquerade IP address sources and preserves source Pod IP addresses. If you need more than two CIDRs, add more entries to the nonMasqueradeCIDRs
list following the same format. At minimum, the nonMasqueradeCIDRs
property should include the node and Pod IP address ranges of your cluster.
SYNC_INTERVAL
: the amount of time after which each ip-masq-agent
Pod checks the contents of the ip-masq-agent
ConfigMap and writes any changes to its local /etc/config/ip-masq-agent
file. Defaults to 60
.
UNIT_OF_TIME
: the unit of time for the resyncInterval. Valid values include s
(for seconds) or ms
(for milliseconds). Defaults to s
.
Set masqLinkLocal
to false (the default) unless you need to enable masquerading for packets sent to link local IPv4 addresses. For more information, see Masquerading to link-local destinations.
nonMasqueradeCIDRs
must at least include the cluster's node and Pod IP address range(s). This is especially important if you enable intranode visibility. If the Pod IP address range(s) are not present in nonMasqueradeCIDRs
and you have intranode visibility turned on, Pods won't be able to properly communicate with each other.Create the ConfigMap resource:
kubectl create configmap ip-masq-agent \
--namespace=kube-system \
--from-file=config=LOCAL_CONFIG_FILE_PATH
Replace LOCAL_CONFIG_FILE_PATH
with the path to the config file you created in the previous step.
ip-masq-agent
DaemonSet expects when reading the ip-masq-agent
ConfigMap to parse it as a volume. In the following example, and for standard ip-masq-agent
DaemonSets, the key is config
. Specifying --from-file=config=LOCAL_CONFIG_FILE_PATH
ensures that the IP masquerade configuration information is associated with the config
key.Describe the ip-masq-agent
ConfigMap in the kube-system
namespace:
kubectl describe configmaps/ip-masq-agent -n kube-system
The output is similar to the following:
Name: ip-masq-agent
Namespace: kube-system
Labels: <none>
Annotations: <none>
Data
====
config:
----
nonMasqueradeCIDRs:
- 198.15.5.92/24
- 10.0.0.0/8
masqLinkLocal: false
resyncInterval: 60s
BinaryData
====
Events: <none>
This output includes the config
parameter with your configuration changes. You can now deploy the ip-masq-agent
DeamonSet.
ip-masq-agent
ConfigMap
You can modify the contents of an existing ip-masq-agent
ConfigMap by completing the following steps:
Open the ConfigMap in a text editor:
kubectl edit configmap ip-masq-agent --namespace=kube-system
Edit the content of the ConfigMap file:
apiVersion: v1
data:
config: |
nonMasqueradeCIDRs:
- CIDR_1
- CIDR_2
masqLinkLocal: false
resyncInterval: SYNC_INTERVALUNIT_OF_TIME
kind: ConfigMap
metadata:
name: ip-masq-agent
namespace: kube-system
Replace the following:
CIDR_1
and CIDR_2
: the IP address ranges in CIDR format. When packets are sent to these destinations, your cluster does not masquerade IP address sources and preserves source Pod IP addresses. If you need more than two CIDRs, add more entries to the nonMasqueradeCIDRs
list following the same format. At minimum, the nonMasqueradeCIDRs
property should include the node and Pod IP address ranges of your cluster.
SYNC_INTERVAL
: the amount of time after which each ip-masq-agent
Pod checks the contents of the ip-masq-agent
ConfigMap and writes any changes to its local /etc/config/ip-masq-agent
file. Defaults to 60
.
UNIT_OF_TIME
: the unit of time for the resyncInterval. Valid values include s
(for seconds) or ms
(for milliseconds). Defaults to s
.
Set masqLinkLocal
to false (the default) unless you need to enable masquerading for packets sent to link local IPv4 addresses. For more information, see Masquerading to link-local destinations.
nonMasqueradeCIDRs
must at least include the cluster's node and Pod IP address range(s). This is especially important if you enable intranode visibility. If the Pod IP address range(s) are not present in nonMasqueradeCIDRs
and you have intranode visibility turned on, Pods won't be able to properly communicate with each other.Describe the ip-masq-agent
ConfigMap in the kube-system
namespace:
kubectl describe configmaps/ip-masq-agent -n kube-system
The output is similar to the following:
Name: ip-masq-agent
Namespace: kube-system
Labels: <none>
Annotations: <none>
Data
====
config:
----
nonMasqueradeCIDRs:
- 198.15.5.92/24
- 10.0.0.0/8
masqLinkLocal: false
resyncInterval: 60s
BinaryData
====
Events: <none>
This output includes the config
parameter which matches the configuration value from the file you created.
ip-masq-agent
DaemonSet
After you create or edit your ip-masq-agent
ConfigMap, deploy the ip-masq-agent
DaemonSet.
Save the following manifest as a YAML file:
apiVersion: apps/v1 kind: DaemonSet metadata: name: ip-masq-agent namespace: kube-system spec: selector: matchLabels: k8s-app: ip-masq-agent template: metadata: labels: k8s-app: ip-masq-agent spec: hostNetwork: true containers: - name: ip-masq-agent image: gke.gcr.io/ip-masq-agent:v2.12.3-gke.4@sha256:b5db41ddaf863b660da330322714f668101482b528829c50c53229d901d11af5 args: - --v=2 - --logtostderr=false - --log_file=/dev/stdout - --log_file_max_size=0 # The masq-chain must be IP-MASQ - --masq-chain=IP-MASQ # To non-masquerade reserved IP ranges by default, # uncomment the following line. # - --nomasq-all-reserved-ranges # Must be set to false when using Dataplane V2. - --random-fully=false securityContext: privileged: false capabilities: drop: ["ALL"] add: ["NET_ADMIN", "NET_RAW"] allowPrivilegeEscalation: false seccompProfile: type: RuntimeDefault volumeMounts: - name: config-volume mountPath: /etc/config volumes: - name: config-volume configMap: name: ip-masq-agent optional: true items: - key: config path: ip-masq-agent tolerations: - effect: NoSchedule operator: Exists - effect: NoExecute operator: Exists - key: "CriticalAddonsOnly" operator: "Exists"
This manifest creates a volume named config-volume
which is mounted as specified by the container's volumeMount.
If you need to edit this manifest, consider the following conditions:
The volume name can be anything but must match the container's volumeMount
name.
The ConfigMap name must match the name of the configMap
referenced in the config-volume
Volume in the Pod.
The name of the chain (--masq-chain
) must be IP-MASQ
. Otherwise, GKE does not override the default masquerading rules.
DaemonSet Pods read from the ip-masq-agent
file. The ip-masq-agent
file content is the value of the config
key in the ConfigMap.
If you use non-masquerade reserved IP ranges by default, uncomment the - --nomasq-all-reserved-ranges
line in the arg
section.
Deploy the DaemonSet:
kubectl apply -f LOCAL_FILE_PATH
Replace LOCAL_FILE_PATH
with the path to the file you created in the previous step.
You can manually update the ip-masq-agent
DaemonSet you created. To learn more, see Updating a DaemonSet.
ip-masq-agent
This section shows you how to delete the ip-masq-agent
DaemonSet and the ip-masq-agent
ConfigMap. Deleting the ip-masq-agent
doesn't revert the existing IP masquerading settings on the nodes.
ip-masq-agent
DaemonSet
If you manually created the ip-masq-agent
DaemonSet, you can delete it by running the following command:
kubectl delete daemonsets ip-masq-agent -n kube-system
Note: If the ip-masq-agent
DaemonSet was deployed automatically, GKE will automatically recreate it. Deleting the ip-masq-agent
ConfigMap
To completely delete the ip-masq-agent
ConfigMap, run the following command:
kubectl delete configmap ip-masq-agent -n kube-system
Troubleshooting
The following sections provide troubleshooting advice.
General troubleshootingThe following steps help you diagnose issues with IP address masquerading:
ip-masq-agent
. If the ConfigMap is not defined, traffic to all default destinations is not masqueraded and preserves the Pod IP address. Traffic to other destinations preserves the node IP address.ip-masq-agent
image specified in the DaemonSet. If the version of the ip-masq-agent
DaemonSet is not the latest, follow the deployment steps to update the DaemonSet.sudo iptables -t nat -L IP-MASQ
command in the impacted node. If the nonMasqueradeCIDRs
defined in the ConfigMap is not showing up on the NAT IP Tables, verify that there are no typos in the config file that was used to create the ConfigMap.While using an IP masquerade agent, you might notice the Pod's source IP address unexpectedly use the node's IP address when your Pods communicate with external destinations.
The issue is caused by a missing or incomplete custom Source Network Address Translation (SNAT) list. When you use an IP masquerade agent, the default SNAT of the cluster is used when the ConfigMap is missing or it doesn't contain a nonMasqueradeCIDRs
list. When a packet leaves a Pod, the default SNAT changes the source IP address from the Pod's IP address to the node's internal IP address. To learn more about SNAT, see IP masquerade agent.
To resolve the issue, configure a custom SNAT list by defining a nonMasqueradeCIDRs
list in the ip-masq-agent
ConfigMap:
Open the ip-masq-agent
ConfigMap:
kubectl edit configmap ip-masq-agent --namespace=kube-system
Review the nonMasqueradeCIDRs
list within the ConfigMap. The nonMasqueradeCIDRs
list should be present and complete. For example:
...
nonMasqueradeCIDRs:
- 35.100.0.0/16
...
If the list is missing or incomplete, add or modify the nonMasqueradeCIDRs
list to include the destination IP ranges for which you want to preserve source Pod IP addresses. This list should also include the addresses that you want to use Cloud NAT.
If you don't want any external traffic to use SNAT, set the nonMasqueradeCIDRs
field to 0.0.0.0/0
. For example:
...
nonMasqueradeCIDRs:
- 0.0.0.0/0
...
When your traffic doesn't use SNAT, all packets sent from Pods retain the Pod's IP address as their source IP address, regardless of destination.
Check the source IP address of outbound packets from your Pods. To check this address, take a packet capture. If that's not feasible, you can check the IP address of the node, which should be the source IP address of outbound packets subject to SNAT, with the following command:
kubectl get nodes -o wide
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4