Ramblings of an aging IT geek
← Ramblings of an aging IT geek
debugging

it was the mtu, it's always the mtu

A connection that worked for small requests and hung on large ones, which is the unmistakable signature of a broken path MTU.

A bug glowing in a terminal

The symptom was the tell, and I still didn't read it for an hour. A new service could talk to the database fine. Small queries, instant. Health checks, green. But any query that returned more than a few kilobytes hung, then eventually timed out. SSH had the same flavour: the login worked, the prompt appeared, and then ls on a big directory froze. Small things fine, big things dead.

That pattern, works small, hangs large, is path MTU discovery quietly failing, every time. Somewhere on the route was a link with a smaller MTU than the endpoints assumed, usually because of a tunnel adding its own header, and the "packet too big" ICMP message that's supposed to fix it was being eaten by a firewall that blocks ICMP because someone once read that ICMP is dangerous. So the sender keeps cheerfully transmitting full-size frames that silently get dropped at the bottleneck, and the connection wedges the moment it tries to send anything that fills a packet.

The confirmation is two pings. One that fits, one that doesn't, with don't-fragment set:

$ ping -M do -s 1400 db.internal
PING db.internal: 1400 data bytes
... ok
$ ping -M do -s 1500 db.internal
ping: local error: message too long, mtu=1450

There it is. The path can take 1450, not 1500, because of a tunnel I'd forgotten was even there. The honest fix is to stop the firewall dropping ICMP fragmentation-needed messages, so PMTU discovery can do its job. The quick fix, when you don't own that firewall, is to clamp the MTU on the interface, or clamp MSS at the gateway, so nothing ever tries to send a packet the path can't carry.

$ sudo ip link set dev eth0 mtu 1450

Connection healed instantly. I've now lost enough hours to this exact shape of bug that it's the first thing I reach for: when small works and large hangs, it's the MTU. It's always the MTU.