Ramblings of an aging IT geek
← Ramblings of an aging IT geek
networking

when packets vanish: an mtu mystery

A homelab tunnel that worked for small requests and silently hung on large ones, traced to an MTU mismatch and broken path MTU discovery.

Network cables fanning out from a switch

SSH worked. ping worked. Pulling a small file worked. Pulling a large one hung dead at exactly the same point every time. That specific shape of failure, small things fine and big things stalling, is MTU until proven otherwise.

I'd put a WireGuard tunnel between two sites. WireGuard adds its own overhead, so the effective MTU inside the tunnel is smaller than the 1500 the interfaces happily advertised. Normally path MTU discovery sorts this out: a router sends back an ICMP "fragmentation needed" and everyone agrees on a smaller size. But something in the path was eating those ICMP messages, so the discovery never completed. Packets that were too big just disappeared into the void, and TCP sat there retransmitting forever.

The confirmation is a one-liner. Ping with the don't-fragment bit set and a payload just under the suspected limit:

ping -M do -s 1400 10.0.2.1   # works
ping -M do -s 1420 10.0.2.1   # silence

That silence at 1420 told me everything. I set the WireGuard interface MTU to 1380, clamped TCP MSS on the tunnel for good measure, and the large transfers came back instantly.

MTU bugs are nasty because nothing logs an error. Connections don't refuse, they just hang, and you waste an hour blaming DNS or the application before you remember that a 1500-byte assumption is a lie the moment you tunnel anything.