Ramblings of an aging IT geek
← Ramblings of an aging IT geek
news

half the internet leaned on one region, and it showed

A reaction to the big AWS us-east-1 outage earlier this month and the uncomfortable reminder of how much rides on a single region.

Tech news on a screen

My feeds spent most of last week chewing over the AWS outage. On 20 October a problem in us-east-1, by AWS's own account tied up with DNS resolution for DynamoDB, rippled outward and took a startling slice of the internet with it. Apps that have nothing obviously to do with each other all went dark at once, which is always the tell that they share a dependency they never mention.

What got me wasn't the outage itself. Big providers have bad days, and on the whole they have far fewer than I would. What got me was the blast radius, and how many services I use turned out to be sitting on the same region without ever saying so. You only find out who your neighbours are when the building catches fire.

I went and looked at my own stuff afterwards, half expecting to feel smug. I didn't, particularly. The services I'd moved home were fine, because they don't depend on us-east-1 to start up. But a couple of things I rely on for the moving-home itself, a tunnel here, a registrar's API there, are absolutely in that blast radius, and I'd just never had cause to notice. That's the trap: the dependency is invisible right up until the day it isn't.

I'm not going to pretend there's a tidy lesson, or that I'm about to architect for multi-region resilience on a personal blog. But it's a fair nudge to actually know where my critical paths land, rather than assuming. Most of the time a single region is fine. It's the other times that write the postmortems.