Ramblings of an aging IT geek
← Ramblings of an aging IT geek
networking

the afternoon i broke dns and then couldn't look up how to fix it

I took down name resolution across the whole house by pointing the resolver at a server that depended on the resolver, and the recovery was entirely manual.

Network cables coiled behind a rack

It's always DNS. Everyone says it, everyone laughs, and then one of us does it to ourselves anyway. Last Sunday it was my turn.

I was tidying up the homelab's resolver setup, moving from a flaky old Pi running dnsmasq to a tidy Pi-hole in a container. Sensible upgrade. I pointed the router's DHCP at the new resolver, restarted, and the whole house went quiet. No web, no streaming, the partner's laptop sulking, the works. The new resolver was a container on a host that pulled its image from a registry by hostname, on a network whose only resolver was now the container that hadn't started yet. Chicken, meet egg.

The worst part is the instinct under pressure is to search for the fix, and you can't, because the thing you'd search with is down. I ended up SSHing in by IP, hardcoding a public resolver in /etc/resolv.conf long enough to pull the image, then starting the container, then putting things back. The proper fix afterwards was the boring one: the container host now has a static, dependency-free resolver of its own, so it can always bootstrap regardless of what the house-wide resolver is doing. Never let the thing that resolves names depend on names to come up. I knew that. I'll know it harder now.