Ramblings of an aging IT geek
← Ramblings of an aging IT geek
networking

the homelab that worked for everything except big files

A debugging post about an MTU mismatch on a homelab VPN that let small traffic through fine but quietly stalled anything large, and how to find it.

A bundle of network cables behind a rack

Everything worked. That's the trap with MTU. SSH worked. Ping worked. Pulling up a web page over the VPN worked. It all worked right up until I tried to copy anything large across the link, at which point it would transfer a few kilobytes, then hang, dead, forever. No error. No reset. Just a progress bar frozen at 4% while I lost an afternoon.

The symptom that should have told me immediately: small things pass, big things stall. That is the signature of an MTU problem, and I knew that, and I still spent two hours checking firewall rules and DNS and routing tables before the penny dropped. MTU is the silent killer precisely because the things you test with first, ping and a quick login, are too small to trigger it.

what was actually happening

The link in question was a WireGuard tunnel between my homelab and a small VPS. WireGuard adds its own header on top of every packet, around 60 bytes of overhead once you account for the UDP and IP wrapping. The host interface had the usual Ethernet MTU of 1500. The tunnel did not have 1500 bytes to play with, it had 1500 minus the overhead, but the machines either side hadn't been told that.

So a full-size 1500-byte packet would arrive at the tunnel, be too big to fit once the WireGuard header was bolted on, and need fragmenting. And here's the killer: those packets had the Don't Fragment bit set, as TCP packets generally do, so they couldn't be fragmented. The correct response is an ICMP "fragmentation needed" message back to the sender telling it to use smaller packets. That mechanism is Path MTU Discovery, and it's supposed to handle exactly this automatically.

It didn't, because somewhere along the path those ICMP messages were being dropped. Plenty of firewalls block ICMP wholesale on the lazy theory that ICMP is "just ping" and therefore optional. It is not optional. Block the wrong ICMP types and you break PMTUD, and when PMTUD breaks you get exactly this: a connection that establishes fine on small packets and then black-holes the moment it tries to send a full-size one. The technical name is a PMTUD black hole. The practical name is "why is my file copy stuck at 4%".

A rack of servers, the quiet scene of the crime

finding it

The test that pins it down in one command is a ping with a fixed payload size and Don't Fragment set. On Linux:

ping -M do -s 1472 10.0.0.2

The 1472 is 1500 minus the 28 bytes of ICMP and IP header, so that's a full-size frame. Run it across the tunnel and it fails with "message too long" or just silently drops, while a smaller size sails through:

ping -M do -s 1372 10.0.0.2   # works
ping -M do -s 1472 10.0.0.2   # hangs or errors

Walk the size down until it starts working, add the overhead back, and you've measured your real path MTU. In my case the magic number was about 1420 bytes of payload, which lined up neatly with WireGuard's documented overhead.

fixing it

Two honest fixes. The proper one is to set the tunnel interface MTU correctly so the kernel never tries to send oversized packets in the first place:

ip link set dev wg0 mtu 1420

WireGuard usually picks a sensible MTU itself, but if you've layered it over something with its own overhead, a VPN inside a tunnel, PPPoE, the maths shifts and you may need to set it by hand.

The other fix, the one that papers over the whole class of problem, is MSS clamping. You tell the router to rewrite the Maximum Segment Size in passing TCP handshakes so both ends agree to use smaller segments from the start, no PMTUD required:

iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN \
  -j TCPMSS --clamp-mss-to-pmtu

That clamp is doing a lot of quiet work on home routers everywhere, which is part of why people rarely meet this problem head-on until they build a tunnel themselves.

The lesson I keep re-learning is the diagnostic one, not the fix. When something works for small operations and dies on large ones, stop checking firewall rules and DNS and go straight to MTU. It hides perfectly behind a working ping, and it'll let you waste an entire afternoon before you remember it exists. Ask me how I know.