MTU mismatch in Docker and OpenStack: how to fix CI/CD failure

How MTU Mismatch in Docker and OpenStack Breaks CI/CD: Diagnose and Fix

A Friday night build failure is often no random glitch—it's usually an MTU mismatch between Docker containers and OpenStack's overlay network. The issue only surfaces with large data transfers, like cloning a repo, while small requests go through fine. This creates a false sense of a working network until real workloads hit.

Why Small Requests Work but Big Ones Fail

The hallmark of this problem is its selectivity. Commands like curl -I or ping that send tiny packets succeed every time. Trouble starts with ops needing big data chunks: git clone, docker pull, artifact downloads. The culprit? Maximum Transmission Unit (MTU)—the largest IP packet size that can transmit without fragmentation.

Standard Ethernet networks use an MTU of 1500 bytes. But in cloud setups like OpenStack, overlay tech like VXLAN or GRE builds isolated networks by wrapping each packet in about 50 bytes of overhead. So even if the physical network handles 1500, your VM packets are left with just 1450 bytes.

Docker defaults to a bridge network with MTU 1500, ignoring infra quirks. Containers build packets to that spec. When they hit the host's 1450 MTU interface, they can't transmit because of the DF (Don't Fragment) flag. Path MTU Discovery should kick in ideally, but firewalls, security groups, or wonky routers often block those "Fragmentation Needed" ICMP messages.

Diagnose the Issue in 30 Seconds

Spot an MTU mismatch with these three commands:

• Check the host network interface MTU:

```

ip link show ens3 | grep mtu

```

In OpenStack, expect mtu 1450.

• Check MTU inside a container:

```

docker run --rm alpine ip link show eth0 | grep mtu

```

Docker defaults to mtu 1500. If it's higher than the host, you've found the problem.

• Run a ping test blocking fragmentation:

```

docker run --rm alpine ping -s 1422 -M do -c 3 gitlab.example.com

docker run --rm alpine ping -s 1423 -M do -c 3 gitlab.example.com

```

The first (1422 = 1450 − 28) should succeed; the second should fail with Message too long.

Step-by-Step Fix

Set MTU in Docker

The core fix: configure Docker to use the right MTU. Edit /etc/docker/daemon.json:

{
  "mtu": 1450
}

Restart the Docker daemon:

sudo systemctl restart docker

Handle Existing Networks

Note: daemon.json changes only affect new networks. Existing ones—especially from Docker Compose—keep the old MTU. Two options:

• Recreate everything:

```

docker compose down && docker compose up -d

```

• Explicitly set MTU in Docker Compose:

```yaml

networks:

app-net:

driver: bridge

driver_opts:

com.docker.network.driver.mtu: 1450

```

Final Steps

Restart GitLab Runner and test:

sudo gitlab-runner restart
docker run --rm alpine ping -s 1422 -M do -c 3 gitlab.example.com

If the test passes, you're good.

Common Pitfalls and Best Practices

• Don't undershoot MTU "just in case". Dropping to 1300 cuts throughput 10-15% from extra packets. Use the exact diagnosed value.
• Automate it. Bake MTU into your infra config (Terraform, Ansible, cloud-init)—no manual tweaks.
• Mind custom networks. daemon.json skips existing ones. Check with docker network inspect.
• Skip Path MTU Discovery hopes. Cloud envs break it with ICMP blocks.

Key Takeaways

• MTU mismatch only bites on big data transfers, making it sneaky to spot.
• OpenStack overlays eat 50 bytes of MTU with headers.
• Docker's default 1500 clashes in clouds.
• Fix: Match Docker MTU to host interface.
• Automate setup in your deployment pipeline.

— Editorial Team