Ramblings of an aging IT geek
← Ramblings of an aging IT geek
personal

the bug i fixed without touching the keyboard

How an hour on the bike, away from the screen entirely, untangled a race condition I'd been staring at all morning.

Coffee and a stack of books

I had a bug this week that refused to be cornered. Intermittent, of course, the worst kind. A service would occasionally return stale data for a request, maybe one in a few thousand, and only under load. I'd added logging, then more logging, then logging to the logging, and the act of observing it seemed to make it shy away. Classic.

By late morning I'd done the thing every engineer does, which is stop reading the code and start arguing with it. I was convinced the cache was at fault. I rewrote a chunk of the invalidation logic, broke two tests, reverted, and sat there with that hot, narrow feeling where you can no longer see the problem because you're too close to it.

So I gave up. Not on the bug, on the sitting. I changed into something with lycra in it, wheeled the bike out, and went for a ride along the canal.

A wide landscape view

There's a stretch about twenty minutes out where the path opens up and you stop thinking about traffic, and somewhere along there, with no laptop in sight, the answer just arrived. It wasn't the cache at all. It was two goroutines sharing a struct I'd assumed was immutable after construction, and one of them was mutating a slice header on a code path I'd genuinely forgotten existed. The "stale" data was a half-written update being read mid-flight.

I didn't even need to test the hypothesis to know it was right. The shape of it fitted every symptom: intermittent, load-dependent, allergic to logging because the extra timing changed the interleaving. I'd been staring straight at it for hours and couldn't see it, and then I looked away and there it was.

The fix took four lines when I got home. A copy at the boundary, and a comment explaining why, so future-me doesn't "optimise" it back out.

I've stopped treating this as a quirk and started treating it as part of the process. Debugging isn't only the typing. A surprising amount of it is the diffusion that happens when you stop demanding an answer and let the back of your brain get on with it. The bike is just where mine happens to live. For someone else it's the shower, or the washing up, or the walk to get milk. The trick is trusting it enough to actually leave the desk, which, when you're stuck and frustrated, feels exactly like quitting. It isn't.