Path MTU – AWS and GCP

Part 1: GCP and AWS MTU

This article focuses on Internet Control Message Protocol (ICMP) and how Maximum Transmission Unit (MTU) affects the end-to-end throughput on a TCP connection. This topic is important because some organisations encounter issues with MTU and throughput when connecting to their hybrid public cloud environments.

MTU is the largest size of a packet (in bytes) that can be transported across a network. The larger the MTU, the greater the data that can be transported in a single packet. Therefore, we can achieve a greater throughput across a TCP connection by using a larger MTU value.

ICMP is an error reporting mechanism. It used by routers that encounter errors while attempting to transmit a packet, to report the error to the host that originated the packet. One of the error messages that a router can relay back to a host is MTU mismatch. We will discuss this further later in the article. First let’s take a look at the diagram we’ll be using in our scenario below.

GCP and AWS Multi-Cloud

The diagram shows our multi-cloud scenario, where GCP is connected to AWS over an IPsec tunnel. GCP has 2 instances in 172.16.10.0/24 subnet. AWS has 2 instances in the 10.0.1.0/24 subnet. To understand more about IPsec VPN configuration between GCP and AWS, please refer to the previous article – IPsec VPN – GCP and AWS.

GitHub Terraform Script

The scenario is implemented in Terraform as shown in this GitHub repository.

GCP MSS and MTU Values

The Maximum Segment Size (MSS) is the largest amount of data that a host can receive in a single TCP segment. The MTU is typically the MSS plus IP header size (20 bytes) and TCP header size (20 bytes).

On gcp-host-1 host, we can see that the MSS is 1420 bytes as shown in output below.

kayode@gcp-host-1:~$ sudo tcpdump -s0 -p -ni eth0 '(ip and ip[20+13] & tcp-syn != 0)'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
19:21:03.993198 IP 172.16.10.2.39171 > 169.254.169.254.80: Flags [S], seq 3101313099, win 28400, options [mss 1420,sackOK,TS val 278899 ecr 0,nop,wscale 7], length 0
19:21:03.993672 IP 169.254.169.254.80 >; 172.16.10.2.39171: Flags [S.], seq 715930112, ack 3101313100, win 65535, options [mss 1420,eol], length 0

 

The corresponding MTU for gcp-host-1 host is 1460 bytes (1420 bytes + 40 bytes). We can confirm that from the MTU on eth0 interface in the outputs below.

kayode@gcp-host-1:~$ ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
 inet 127.0.0.1/8 scope host lo
 valid_lft forever preferred_lft forever
 inet6 ::1/128 scope host
 valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1460 qdisc pfifo_fast state UP group default qlen 1000
 link/ether 42:01:ac:10:0a:02 brd ff:ff:ff:ff:ff:ff
 inet 172.16.10.2/32 brd 172.16.10.2 scope global eth0
 valid_lft forever preferred_lft forever
 inet6 fe80::4001:acff:fe10:a02/64 scope link
 valid_lft forever preferred_lft forever
kayode@gcp-host-1:~$ ip link show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1460 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
 link/ether 42:01:ac:10:0a:02 brd ff:ff:ff:ff:ff:ff

 

Similar values for MSS and MTU should be expected for gcp-host-2.

AWS MSS and MTU Values

On aws-host-1 host, we can see that the MSS is 8961 bytes as shown in output below.

[ec2-user@ip-10-0-1-113 ~]$ sudo tcpdump -s0 -p -ni eth0 '(ip and ip[20+13]& tcp-syn != 0)'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
19:26:50.108077 IP 10.0.1.113.55564 > 169.254.169.254.http: Flags [S], seq 1511580826, win 26883, options [mss 8961,sackOK,TS val 579792 ecr 0,nop,wscale 7], length 0
19:26:50.108356 IP 169.254.169.254.http > 10.0.1.113.55564: Flags [S.], seq 1920904006, ack 1511580827, win 17898, options [mss 8961,sackOK,TS val 1159445044 ecr 579792,nop,wscale 7], length 0
19:26:50.109607 IP 10.0.1.113.55566 > 169.254.169.254.http: Flags [S], seq 1379098157, win 26883, options [mss 8961,sackOK,TS val 579792 ecr 0,nop,wscale 7], length 0
19:26:50.109801 IP 169.254.169.254.http > 10.0.1.113.55566: Flags [S.], seq 4141905875, ack 1379098158, win 17898, options [mss 8961,sackOK,TS val 1159445044 ecr 579792,nop,wscale 7], length 0

 

The corresponding MTU for aws-host-1 host is 9001 bytes (8961 bytes + 40 bytes). We can confirm that from the MTU on eth0 interface in the outputs below.

[ec2-user@ip-10-0-1-113 ~]$ ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9001
        inet 10.0.1.113  netmask 255.255.255.0  broadcast 10.0.1.255
        inet6 fe80::400:9cff:fe96:5c8e  prefixlen 64  scopeid 0x20<link>
        ether 06:00:9c:96:5c:8e  txqueuelen 1000  (Ethernet)
        RX packets 1006  bytes 122025 (119.1 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1230  bytes 126976 (124.0 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1  (Local Loopback)
        RX packets 16  bytes 1296 (1.2 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 16  bytes 1296 (1.2 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
[ec2-user@ip-10-0-1-113 ~]$ ip link show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc pfifo_fast state UP mode DEFAULT qlen 1000
 link/ether 06:00:9c:96:5c:8e brd ff:ff:ff:ff:ff:ff

Similar values for MSS and MTU should be expected for aws-host-2. 

We can observe that AWS support jumbo frames (greater than 1500 bytes of data per packet). With jumbo frames, fewer packets are needed to transport the same amount of data when compare to the normal 1500 MTU packets. For more information on AWS EC2 instance MTU, refer to the AWS documentation.

Next, we now look at how path MTU is negotiated end-to-end when a host in GCP is communicating with a host in AWS – in Part 2: MTU and ICMP.

Leave a Reply

Your email address will not be published. Required fields are marked *