Ramblings of an aging IT geek
← Ramblings of an aging IT geek
debugging

tcpdump Saved Me Again

A reminder that when two services disagree about what's on the wire, the packet capture is the only witness who isn't lying.

A terminal full of packet capture output

A service was returning 401s to its own health check. The auth team swore the token was being sent. The app team swore it wasn't arriving. Both had logs to prove their point, both logs were technically true, and we'd burned half a morning on a meeting where everyone read their own dashboard at each other.

So I stopped reading logs and started reading the wire:

tcpdump -i any -A -s0 'tcp port 8080 and host 10.0.4.12'

There it was, in plain ASCII. The Authorization header was present, exactly as the auth team claimed. It was also lowercase: authorization. A proxy in the middle had normalised the header name, and the receiving app was doing a case-sensitive map lookup it should never have been doing. Nobody was lying. Everybody was right. The packets just didn't care about anyone's feelings.

tcpdump has no opinion. It doesn't have a log level you forgot to turn up, it doesn't buffer, it doesn't summarise the interesting bit out of existence. When two systems disagree about reality, it's the only neutral witness in the room. I reach for it more often than any debugger, and it has never once let me down. Learn the filter syntax. It pays for itself the first time someone confidently tells you something the packets flatly contradict.