log4shell, and the long week of finding out where java was hiding

If your feeds look anything like mine this week, they are wall to wall with one thing: Log4Shell. CVE-2021-44228, the remote code execution flaw in Log4j 2, landed late last week and has not let go since. Apache have shipped fixes, a follow-up CVE has already appeared because the first patch did not fully close it, and a sizeable chunk of the industry has spent the weekend doing the least glamorous job in security: working out where on earth the vulnerable library actually is.

I will spare you the deep technical write-up, plenty of better ones exist already. The short version is that Log4j's message lookup feature would, on seeing a string like ${jndi:ldap://...} in something it logged, go off and fetch and execute remote code via JNDI. The horror is in that sentence. The thing that logs your input executes your input. And logging is everywhere, so the attack surface is "anywhere user-controlled text reaches a log line", which is to say everywhere.

the genuinely uncomfortable bit

The vulnerability itself is bad enough. What made this week long was the second-order problem: nobody actually knows their dependency tree.

Log4j is not usually something you chose. It is something three layers down, pulled in transitively by a framework, bundled into a vendor appliance, baked into a container image you have not rebuilt since spring. The CVE was easy to read about. The hard question, the one that ate the actual hours, was: am I running it, and where?

We went looking. Not "check the obvious Java services" looking, but properly looking, because the obvious ones were the easy ones. The grep that genuinely worried me was the one that turned things up I had forgotten were Java at all: a monitoring agent, a bit of middleware, a build tool, a piece of network kit whose firmware nobody had thought about in years.

A rough sweep of what I reached for, in escalating order of dread:

# find the jars on disk
find / -name '*.jar' 2>/dev/null \
  | while read j; do
      unzip -l "$j" 2>/dev/null \
        | grep -q 'JndiLookup.class' && echo "VULN-ish: $j"
    done

# what's actually loaded by running JVMs
for pid in $(pgrep -f java); do
  ls -l /proc/$pid/root 2>/dev/null
  cat /proc/$pid/maps 2>/dev/null | grep -i log4j
done

That second loop matters more than the first, because the jar sitting on disk is not necessarily the one a running process loaded, and a process that has been up for ninety days is running whatever it found ninety days ago. Restarting after patching is not optional here.

mitigation, in order of how much I trust it

The advice shifted under our feet over the weekend as people understood the flaw better, which is itself a useful reminder not to treat the first mitigation you read as gospel.

1. Upgrade Log4j. This is the real fix. Get to the patched 2.x line. Everything else is a stopgap.
2. Remove the JndiLookup class from the jar where you genuinely cannot upgrade. Crude, effective, and at least it is local to the artefact rather than a runtime flag.
3. The log4j2.formatMsgNoLookups flag. Floated early, useful in a pinch, but I would not rely on it as a permanent answer, especially on older 2.x versions where its coverage is incomplete.

The ranking matters. A flag you set in a hurry on Friday is a thing you will forget you set by January, and a stopgap you forget is just a vulnerability with extra steps.

what I am actually taking from this

Two things, neither of them about Java specifically.

First, a software bill of materials stops being a compliance box-tick the moment something like this lands. The teams who had a real inventory of what they shipped spent this week patching. The teams who did not spent it discovering. I know which side I want to be on next time, and there will be a next time.

Second, this is the open-source maintenance burden made painfully visible. Log4j is maintained by a small number of volunteers, and the entire industry leant its full weight on it for one terrible week and then, presumably, went quiet again. The people who fixed this, fast, under enormous pressure, mostly do it for free. That is worth saying plainly, and worth remembering when the next "how did we all depend on this?" moment arrives.

For now: patch, restart, verify the running process and not just the file on disk, and write down what you found. The feeds will move on to something else by the weekend. Your dependency tree will not have changed, only your knowledge of it. Use the knowledge while you still have the fright.