Ramblings of an aging IT geek
← Ramblings of an aging IT geek
news

another log4j point release, and what that month taught me

Reflecting on the run of Log4j patches that closed out 2021 and what the whole episode revealed about how little we know our own dependencies.

A wall of tech news headlines

My feeds this morning are, once again, all Log4j. We've now had a string of releases off the back of the original Log4Shell disclosure in early December, the JNDI lookup hole that let an attacker run code by getting a crafted string logged. The patches kept coming through Christmas: the first fix turned out not to be the end of it, then there was the denial-of-service follow-up, and the Apache team have been shipping point releases to mop up the edges ever since. I'm not going to quote exact CVE numbers from memory because by now there are several and I'd get one wrong, but the shape of it is clear enough.

What I keep coming back to isn't the bug itself. It's that for the first day or two, an enormous number of teams genuinely could not answer "are we affected?" Not because they were careless, but because Log4j was three layers down inside something they'd never thought about. It arrived as a transitive dependency of a library of a framework. Nobody put it there on purpose. It was just always there, doing its job quietly, until it wasn't.

A city skyline at dusk

I spent a chunk of December doing the unglamorous work: building an actual inventory. Not "what did we choose to depend on", but "what is genuinely on disk and on the classpath in production". Those are very different lists, and the gap between them was the whole problem. The teams who came out of this calmly were, almost without exception, the ones who already had a usable software bill of materials and could grep it. The ones who didn't spent the first 48 hours doing archaeology under pressure, which is the worst possible time to learn your build.

So the lesson I'm taking into the new year is unglamorous and a bit boring, which is usually how you know it's the real one. Know what you ship. Generate an SBOM as part of the build, store it next to the artefact, and make "which versions of X are running where" a query rather than an expedition. I'd been meaning to get round to that for ages. Log4Shell turned "meaning to" into "did, over the holidays, slightly resentfully".

The other thing I'll say plainly: the Apache Logging team did good, fast, repeated work on this, under a level of scrutiny most of us would crumble under, almost entirely as volunteers. It's easy to dunk on a dependency when it breaks. It's worth remembering that the thing breaking was free, and the people fixing it over their own Christmas weren't being paid by the companies depending on it. That imbalance is the actual story, and it didn't end on the third of January.