Some network faults announce themselves. The cable's out, the switch is dead, the link light is off, you fix it, you move on. And then there's MTU, which doesn't break anything so much as make a small, specific subset of things mysteriously not work whilst everything around them is perfectly fine. It is the silent killer, and last weekend it came for my VPN.
the symptom that made no sense
I'd set up a WireGuard tunnel between my homelab and a small VPS, nothing exotic, so I could reach things at home from outside without exposing them. It came up immediately. Handshake completed, wg show looked healthy, ping flew back and forth with tiny latency. Job done, I thought, and went to actually use it.
ssh connected and then froze the moment anything substantial came back. I could log in, the prompt appeared, and then ls of a big directory would hang forever. apt update hung. Cloning a git repo hung after the initial chatter. But ping worked. curl of a small page worked. dig worked. Anything small worked, anything large died.
That pattern, small works and large hangs, is one of the most useful diagnostic shapes in all of networking, because it points almost straight at one thing. It isn't routing, because the small packets are routing fine. It isn't DNS, because names resolve. It is the size of the packets. It is MTU.
why size breaks things
A quick refresher, because the mechanism is the whole point. Every link has a maximum transmission unit, the largest packet it'll carry. Standard Ethernet is 1500 bytes. The moment you wrap traffic inside a tunnel, you add headers, and WireGuard's encapsulation eats 60-odd bytes off the top. So a packet that's the full 1500 going into the tunnel is now too big to fit down the tunnel as a single packet once WireGuard has wrapped it.
Normally the network handles this gracefully. The router that can't fit the packet sends back an ICMP "fragmentation needed" message saying, in effect, "too big, the most I'll take is 1420", and the sender shrinks its packets. This is Path MTU Discovery, and when it works you never even know it happened.
It works right up until somebody, somewhere, blocks ICMP. And blocking ICMP is depressingly common, because somebody once read that "ping is a security risk" and dropped all of it with a firewall rule, the helpful "fragmentation needed" messages included. When that happens, the oversized packet is silently dropped and no message comes back. The sender keeps cheerfully transmitting packets that vanish into nothing. This is a PMTUD black hole, and it produces exactly my symptom: the small packets fit and arrive, the large ones are too big, get dropped, and there's no error to tell anyone why. The connection just hangs.
confirming it
You don't have to guess. ping with a fixed size and the don't-fragment bit set will tell you precisely where the ceiling is. On Linux:
ping -M do -s 1372 10.0.0.1 # 1372 + 28 bytes overhead = 1400, works
ping -M do -s 1472 10.0.0.1 # = 1500, "message too long" or just lost
You walk the size up until packets stop getting through, and the boundary is your real path MTU. In my case anything over about 1400 bytes through the tunnel vanished. That confirmed it beyond doubt: the tunnel could carry small packets and was eating large ones, exactly as the theory predicted.
the fix
Two ways to fix this, and I did both, belt and braces.
The proper fix is to tell the tunnel interface its real MTU, so the systems either side never try to send packets too big to fit in the first place. For WireGuard you set it right on the interface:
[Interface]
MTU = 1420
Set it on both ends, bring the tunnel back up, and now the kernel knows the tunnel's true size and sizes its packets accordingly. The black hole problem disappears because nothing oversized is ever generated to fall into it.
The second fix is a wonderful, slightly grubby trick called MSS clamping. TCP negotiates a maximum segment size at the start of every connection, and you can have your router rewrite that negotiation on the fly to a value that's guaranteed to fit, regardless of what either end thinks the MTU is. It's the safety net for the PMTUD black hole, forcing the right packet size even when ICMP is being eaten.
iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN \
-j TCPMSS --clamp-mss-to-pmtu
That --clamp-mss-to-pmtu does the maths for you against the outgoing interface's MTU. With the tunnel MTU set correctly and MSS clamping in place as a backstop, ssh stopped hanging, apt update ran, and the git clone finished. The tunnel went from "technically up but useless" to genuinely working.
the lesson, again
I've been bitten by MTU more than once, which is embarrassing for someone who really ought to recognise the pattern by now. So let me write the pattern down where I'll find it next time.
If small things work and large things hang, especially across a tunnel, VPN, or anything that adds encapsulation, suspect MTU before anything else. Don't go reading application logs. Don't blame the server. Reach for ping -M do with a big payload and find the ceiling. And the reason it's so hard to spot is that it presents as an application problem, the slow ssh, the stuck download, when it's really a quiet truth two layers down: a packet that was too big to fit, dropped without a word, by a network that thought it was helping by blocking ICMP. The silent killer earns the name. Nothing logs an error. Nothing pages. It just doesn't work, for some things, sometimes, until you finally remember to check the size.