Ramblings of an aging IT geek
← Ramblings of an aging IT geek
news

a bug in a stack you have never heard of, in a billion devices

Reflections on the Ripple20 disclosures in the Treck embedded TCP/IP stack, and what a deeply buried supply-chain dependency means for things you cannot patch.

A city skyline at dusk, lights coming on across the buildings

The security story making the rounds this June is a set of vulnerabilities, disclosed by the research group at JSOF and being called Ripple20, in an embedded TCP/IP stack from a company called Treck. If you have never heard of Treck, that is rather the point. Their networking code has been licensed and embedded into devices for the better part of twenty years, and the estimates being thrown around put the affected device count in the hundreds of millions across printers, industrial controllers, medical equipment, power gear and the general sprawl of things that quietly speak IP.

I have spent the week thinking less about the specific bugs, which range from information leaks up to remote code execution, and more about the shape of the problem. This is a supply-chain story dressed as a vulnerability story. A single small vendor wrote some networking code a long time ago. It was good enough and cheap enough to license, so it ended up inside other companies' products, which themselves got rebadged and resold and built into yet larger systems. By the time you trace it all the way down, nobody at the top of the stack knows the Treck code is in there at all.

A dense urban grid seen from above, every building a device

That is the genuinely uncomfortable bit. The vulnerability is not the scary part. We patch vulnerabilities all the time; it is the job. The scary part is that for a huge fraction of the affected devices, there is no patch path. The original manufacturer may have folded, or moved on, or never built an update mechanism in the first place because the device was sold as an appliance that just works. The code is frozen into firmware on a controller bolted to a wall in a building nobody remembers commissioning. You cannot apt upgrade a fifteen-year-old industrial sensor.

the part that maps onto my own work

I do not ship embedded firmware, so it would be easy to read this as someone else's problem. It is not, and the reason is uncomfortable when I look honestly at my own dependency trees.

Every service I run pulls in libraries, which pull in libraries, which pull in libraries. I can name my direct dependencies. I would struggle, off the top of my head, to name everything three levels down. When a serious bug lands in something popular and low-level, the first question is always the same and always harder than it should be: am I even affected? You cannot patch what you cannot find, and most of us cannot quickly produce a complete, accurate inventory of what is actually running in our own stacks.

The difference is purely that I have an upgrade path and the firmware folks often do not. I can bump a version and redeploy. But the underlying disease is the same: a transitive dependency I never consciously chose, buried deep enough that I do not think about it until it is on the front page.

what i am actually going to do about it

Nothing dramatic, but a few concrete things, because despair is not a remediation plan.

  • Get a real software bill of materials for the things I run, generated automatically, so "am I affected" becomes a query and not an archaeology dig.
  • Treat the transitive tree as a first-class concern in dependency scanning, not just the top-level packages I typed into a manifest.
  • For the genuinely un-updatable devices on my own network, the printers and the random IoT gadgets, assume they are compromised and put them somewhere they can do little harm. Network segmentation is the firmware patch you can actually apply yourself.

Ripple20 will be patched, slowly, where it can be patched at all, and forgotten by most people inside a month. The thing worth keeping is the lesson underneath it. The most dangerous dependency is not the one with the bug. It is the one you did not know you had. Most of the work of security is just the unglamorous business of knowing what you are actually running, and almost none of us are as good at that as we tell ourselves we are.

A city at night, countless lit windows, most of them unaccounted for