It started, as these things do, with me tidying up. I'd been running dnsmasq on a little box in the corner for ages, handing out DHCP and resolving the handful of internal names I care about. On Sunday I decided to move it onto a different machine, did the migration, and confidently rebooted the old one to repurpose it. Then the entire house lost the internet.
Not lost as in disconnected. The link was fine, pings to 8.8.8.8 went straight through. But every device had been handed exactly one DNS server by DHCP, the old box, and that box was now busy being reinstalled. Nothing could resolve a name. My partner's laptop, the TV, my phone, all of them sat there spinning. The cruel part is that the new resolver was up and working perfectly. Nobody was pointed at it.
The fix took two minutes once I'd stopped swearing: hand out a second resolver in the DHCP options so there's always a fallback. I'd genuinely never bothered, because the single one had never failed, and a thing that never fails is a thing you forget is a single point of failure. Classic.
The lesson isn't really about DNS. It's that I'd built something with no redundancy and then performed surgery on it during the one window where everyone was home and online. Now there are two resolvers, on two boxes, and the DHCP lease lists both. Belt and braces, for the price of one config line I should have written years ago.