Kubernetes IP-MASQ-AGENT

The video below shows how to configure and test ip-masq-agent in Google Kubernetes Engine (GKE). Detailed information on how the ip-masq-agent works can be found in GCP IP masquerade agent documentation.

 

Scenario

The diagram below shows the setup being tested. 

  • IP Masquerading in GKE

We have a GKE cluster with the following CIDR allocation:

  • node-range = 10.0.4.0/22
  • svc-range = 10.0.32.0/20
  • pod-range = 10.4.0.0/14

The GCP region has a VPN tunnel connecting the cluster to an on-premises network. The on-premises network consists of subnets:

  • RFC 1918 = 172.16.11.0/24
  • RFC 6598 = 100.64.0.0/24

 

Default IP Masquerade on GKE node

First, let’s see the default IP masquerade iptables configuration for a GKE node. We do this by establishing an SSH session into the node and then launching the Container Optimized OS (CoS) toolbox.

Next we display the default iptables masquerade rules:

$:~# iptables -t nat -L IP-MASQ
Chain IP-MASQ (2 references)
target     prot opt source       destination         
RETURN     all  --  anywhere     link-local/16    /* ip-masq-agent: local traffic is not subject to MASQUERADE */
RETURN     all  --  anywhere     10.0.0.0/8       /* ip-masq-agent: local traffic is not subject to MASQUERADE */
RETURN     all  --  anywhere     172.16.0.0/12    /* ip-masq-agent: local traffic is not subject to MASQUERADE */
RETURN     all  --  anywhere     192.168.0.0/16   /* ip-masq-agent: local traffic is not subject to MASQUERADE */
MASQUERADE  all  --  anywhere    anywhere         /* ip-masq-agent: outbound traffic is subject to MASQUERADE (must be last in chain) */

By default, when a pod attempts to reach link local address (169.254.0.0/16) or RFC 1918 IP address, the pod IP address is not masqueraded behind the node IP. The node IP address is masqueraded for traffic going out to any other destination including RFC 6598.

Now we connect to our container with an IP address as 10.4.0.80 and ping the on-premises servers.

We can see that traffic to the server in 100.64.0.0/24 subnet is masqueraded behind the node IP address (10.0.4.28).

:~$ sudo tcpdump -n icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
19:28:37.849340 IP 10.0.4.28 > 100.64.10.158: ICMP echo request, id 2094, seq 4, length 64
19:28:37.849371 IP 100.64.10.158 > 10.0.4.28: ICMP echo reply, id 2094, seq 4, length 64

And that traffic to the server in 172.16.11.0/24 subnet is not masqueraded.

:~$ sudo tcpdump -n icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
19:28:37.730776 IP 10.4.0.80 > 172.16.11.10: ICMP echo request, id 2094, seq 1, length 64
19:28:37.730802 IP 172.16.11.10 > 10.4.0.80: ICMP echo reply, id 2094, seq 1, length 64

IP Masquerade Enabled

Let’s assume that we want to disable masquerading for the RFC 6598 (100.64.0.0/10) destinations. To do this, we configure IP masquerade by creating a ConfigMap yaml named config; with the following contents

apiVersion: v1
kind: ConfigMap
metadata:
  name: ip-masq-agent
  namespace: kube-system
data:
  config: |
    nonMasqueradeCIDRs:
    - 10.0.0.0/8
    - 172.16.0.0/12
    - 192.168.0.0/16
    - 100.64.0.0/10
    resyncInterval: 60s
    masqLinkLocal: false

In the ConfigMap, we included the RFC 6598 range as part of the CIDRs that will not be masqueraded. Next, we deploy our ConfigMap and then verify the iptables on the GKE node

$:~# iptables -t nat -L IP-MASQ
Chain IP-MASQ (2 references)
target     prot opt source       destination         
RETURN     all  --  anywhere     link-local/16   /* ip-masq-agent: local traffic is not subject to MASQUERADE */
RETURN     all  --  anywhere     10.0.0.0/8      /* ip-masq-agent: local traffic is not subject to MASQUERADE */
RETURN     all  --  anywhere     172.16.0.0/12   /* ip-masq-agent: local traffic is not subject to MASQUERADE */
RETURN     all  --  anywhere     192.168.0.0/16  /* ip-masq-agent: local traffic is not subject to MASQUERADE */
RETURN     all  --  anywhere     100.64.0.0/10   /* ip-masq-agent: local traffic is not subject to MASQUERADE */
MASQUERADE  all  --  anywhere    anywhere        /* ip-masq-agent: outbound traffic is subject to MASQUERADE (must be last in chain) */

And lastly, we ping test to the servers in the on-premises subnets again.

We can now see that traffic to the server in 100.64.0.0/24 subnet is no longer masqueraded behind the node IP address (10.0.4.28). The pod IP address (10.4.0.80) is seen as the source address.

:~$ sudo tcpdump -n icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
20:20:26.718697 IP 10.4.0.80 > 100.64.10.158: ICMP echo request, id 2122, seq 4, length 64
20:20:26.718726 IP 100.64.10.158 > 10.4.0.80: ICMP echo reply, id 2122, seq 4, length 64

One Reply to “Kubernetes IP-MASQ-AGENT”

Leave a Reply

Your email address will not be published. Required fields are marked *