Linux kernel error sends corrupted TCP / IP packets to Mesos, Kubernetes and Docker containers / ua-hosting.company's blog

And it was discovered on Twitter

The Linux kernel has an error caused by containers. To avoid checking TCP checksums, containers use veth devices (such as IPv6 Docker, Kubernetes, Google Container Engine, and Mesos) for network routing. This leads to the fact that in some cases applications mistakenly receive corrupted data, as happens with faulty network equipment. We checked that this error appeared at least three years ago and still “sits” in the kernels. Our patch has been tested and introduced into the kernel, and currently provides retro-support for stable release 3.14 in various distributions (such as Suse and Canonical). If you use containers in your system, I recommend that you use this patch or install the kernel with it when it becomes available.

Note: this does not apply to NAT networks used by default for Docker, since the Google Container Engine is practically protected from hardware errors by its virtualized network. Jake Bower also believes that this error is very similar to the Pager Duty error discovered earlier.

How it all started

One weekend in November, a group of Twitter support engineers scanned the logs. Each non-working application generated "incredible" errors in the form of strings of strange characters or omissions of required fields. The relationship between these errors was not obvious due to the nature of Twitter's distributed architecture. The whole situation was complicated by the fact that in any distributed system the data, once corrupted, can cause errors for quite some time (they are stored in caches, written to disks, in journals, etc.).

After a day of continuous work on troubleshooting at the application level, the team managed to localize the problem to individual rack racks. Engineers determined that a significant increase in the number of TCP checksum errors detected occurred immediately before they reached the destination. It seemed that this result “freed” the software: the application could cause network congestion, and not packet damage!


As soon as these racks were turned off, application errors disappeared. Of course, data corruption at the network level can occur for many reasons: in the most bizarre way switches can fail, wires can be damaged, power can often fail, etc. Checksums in TCP and IP are designed to protect against such malfunctions. Indeed, statistics collected on the equipment showed that errors were present. So, why did a variety of applications start to crash?

After isolating certain switches, attempts were made to reproduce these errors (mainly due to the enormous work of senior reliability engineer Brian Martin). It turned out that it is relatively simple to reproduce damaged data by sending a number of data packets to these racks. On some switches, up to 10% of packets sent were corrupted. However, damage was always detected by the kernel using TCP checksums (messages like TcpInCsumErrors netstat-a), and never reached the applications. (On Linux, using the undocumented SO_NO_CHECK option, you can send IPv4 UDP packets with checksums disabled - in this case, we can see that the packet is damaged.)

Evan Jones suggested that corrupted data has the correct TCP checksum. If in the same 16-bit word two bits are mirrored “flipped” (for example, 0 → 1 and 1 → 0), their influence on the TCP checksum will cancel each other out (TCP checksum is a simple addition of bits). Although the data corruption in the message (32 bytes modulo) always consisted of a change in the same bit position, the fact that there was a “sticky bit” (0 → 1) precluding this possibility. On the other hand, since the checksum itself is inverted before being saved, the “flip” of the bit in the checksum together with the “flip” of the bit in the data can cancel each other out. However, the bit position that we observed changed could not touch the TCP checksum, and therefore this was not possible.

Soon, the team realized that our tests were carried out on a standard Linux system, and most of the services on Twitter run on Mesos, which uses Linux containers to isolate various applications. In particular, virtual Ethernet devices (veth devices) were created in the Twitter configuration and all packets for applications are transmitted through them. Of course, after launching our test application in the Mesos container, corrupted data immediately appeared through the TCP connection, despite the fact that the TCP checksums were incorrect (and were detected as incorrect: the number of TcpInCsumErrors increased). Someone suggested changing the “checksum offloading” settings on a virtual Ethernet device, which solved the problem, leading to the corresponding removal of corrupted data.

Now we have a serious task. The Mesos Twitter team quickly relayed information to the Mesos open source project and changed the settings on all Twitter working containers.

Dig further

When Evan and I discussed this error, we decided that since the TCP / IP protocol was violated by the OS, it was possible that it was not the result of Mesos misconfiguration, but most likely the result of a previously undetected error in the kernel network stack.

To continue this investigation, we created the simplest test system possible:

1. A simple client that opens a socket and sends a very simple, long message once per second.
2. a simple server (we actually used a network computer in listening mode), which listens on the socket and when it receives information, displays it on the screen.
3. a network device, a transmission controller that allows the user to arbitrarily damage packets before they are sent over the wire.
4. Once connecting the client and server, we used a network device to corrupt all packets sent in 10 seconds.

We controlled the client from one machine, and the server from another. Computers connected through a switch. When we launched the test system without containers, it behaved exactly as expected. No alerts for damaged packages have been received. In fact, the ten seconds that the corrupted data was being transmitted, we did not receive any messages at all. After the client stopped corrupting packets, all 10 messages (which were not delivered) immediately arrived. This confirmed that TCP on Linux without containers works as it should: bad TCP packets are delayed and constantly retransmitted until they can be received without errors.


So this should work in theory: corrupted data is not delivered; TCP retransmits them.

Linux and containers

In this place it is useful (as we did when diagnosing the problem) to go back and briefly describe how the network stack works in the Linux container environment. The containers were designed to allow user-serving applications to work together on computers and thus provide many advantages of a virtualized environment (reduce or completely eliminate the possibility of applications affecting each other, provide the ability for multiple applications to work in different environments or with different libraries) at a lower cost virtualization. Ideally, all objects that conflict with each other are isolated. Examples include disk request queues, caches, and network operation.

On Linux, veth devices are used to isolate some containers running on a computer from others. The Linux network stack is quite complex, but essentially a veth device is an interface that, from the user's point of view, should look exactly like a “regular” Ethernet device.

To create a container with a virtual Ethernet device, you must (first) create a container, (second) create veth, (third) bind one end of veth to the container, (fourth) assign an IP address for veth and (fifth) configure routing, as a rule , using Linux Traffic Control, so that the container can receive and send packets.

Why is all this virtual economy "falling"

We re-created the test system described above, except that now the server was working inside the container. However, when we launched it, we saw the exact opposite picture: the damaged data was not delayed, but delivered to applications! We reproduced the error on a very simple test system (two machines on the table and two very simple programs).


Damaged data is delivered to the application: look at the window on the left!

We were also able to reproduce this test system on cloud platforms. Kubernetes configuration by default shows the same picture (for example, when using the Google Container Engine). The default configuration of Docker (with NAT) is secure, but the configuration of Docker on IPv6 is not.

We fix the error

After studying the network code of the kernel, it became clear that this error was in the kernel veth module. In the kernel, in packets that come from real network devices, ip_summed is set to CHECKSUM_UNNECESSARY if the hardware checked the checksums, or CHECKSUM_NONE if the packet is bad or there was no way to check it.

The code in veth.c replaced CHECKSUM_NONE with CHECKSUM_UNNECESSARY. This led to the fact that the checksums that were supposed to be verified and rejected by the software were simply silently ignored. After removing this code, packets began to be transferred from one stack to another unchanged (as expected, tcpdump showed incorrect checksums on both sides), and were correctly delivered (or delayed) to applications. We did not check all possible network configurations, only a few common ones, such as the Bridge container connection using NAT between the host and the container, and routing from real devices to containers. We have effectively implemented this into Twitter (disabling the checksum calculation at reception and unloading the veth devices).

We don’t know exactly why the code was written this way, but we believe that this is an optimization attempt. Often veth devices are used to connect containers on the same physical machine. It is logical that packets transmitted between containers (or virtual machines) within the same physical host should not calculate or check checksums: the only possible source of damage is the host's own RAM, since packets are never transmitted over the wire. Unfortunately, this optimization does not work even as intended: for locally transmitted packets, the ip_summed parameter is set to CHECKSUM_PARTIAL, and not to CHECKSUM_NONE.

This code was written for the first driver release (commit e314dbdc1c0dc6a548ecf [NET]: Virtual ethernet device driver). The release of Commit 0b7967503dc97864f283a net / veth: Fix packet checksumming (December 2010) set this for packets created locally and transmitted to real devices without changing CHECKSUM_PARTIAL. However, this problem is still not resolved for packets transmitted from real devices.

The following is a patch for the kernel:

diff — git a/drivers/net/veth.c b/drivers/net/veth.c
index 0ef4a5a..ba21d07 100644
— — a/drivers/net/veth.c
+++ b/drivers/net/veth.c
@@ -117,12 +117,6 @@ static netdev_tx_t veth_xmit(struct sk_buff *skb, struct net_device *dev)
kfree_skb(skb);
goto drop;
}
- /* don’t change ip_summed == CHECKSUM_PARTIAL, as that
- * will cause bad checksum on forwarded packets
- */
- if (skb->ip_summed == CHECKSUM_NONE &&
- rcv->features & NETIF_F_RXCSUM)
- skb->ip_summed = CHECKSUM_UNNECESSARY;

if (likely(dev_forward_skb(rcv, skb) == NET_RX_SUCCESS)) {
struct pcpu_vstats *stats = this_cpu_ptr(dev->vstats);


conclusions

All in all, I am really impressed with the netdev group and Linux kernel experts. The code inspection was quick enough, our patch was connected for several weeks, and a month later it provided retro support for the old, stable version (3.14 +) with various kernel distributions (Canonical, Suse). With all the advantages of the container environment, it really came as a big surprise to us that this error lasted for many years unnoticed. I wonder how many failures and other unpredictable behavior of application programs could be caused by it!