the box that was busy doing nothing

A server that should have been idle was sitting at a steady 15% CPU and refusing to explain itself. No deployment, no cron job, no obvious workload. top showed the usage but pinned it on nothing in particular, which is the worst kind of answer because it means the cost is below the resolution of the tools I reach for first.

So I went to perf top, which samples where the CPU actually spends its time, kernel and userspace both, and prints a live ranked list of functions. On Debian you want the matching linux-perf package and root:

perf top -g

The -g gives you call graphs, which is the difference between "something in the network stack" and "this specific path, called from here". Within a few seconds the top of the list was dominated by softirq handling and a hash lookup deep in the conntrack code. That was the clue.

The box was a small NAT gateway, and connection tracking was doing real work on every packet. Nothing pathological by itself, but I'd been sloppy: a noisy monitoring agent on the LAN was opening and tearing down a connection every few hundred milliseconds to half a dozen endpoints, and each of those churned the conntrack table. Multiply that across a fleet of similar agents and the kernel was spending genuine cycles inserting and expiring entries. The userspace processes looked idle because they were idle. The work was all in the kernel, which is exactly the work top is bad at attributing.

Two fixes. First, I told the monitoring agent to reuse connections instead of slamming them open and shut, which is just good manners. Second, for the genuinely stateless flows passing through the gateway I added a notrack rule so conntrack stops bothering with traffic that doesn't need it:

table ip raw {
  chain prerouting {
    type filter hook prerouting priority -300;
    ip daddr 10.0.0.0/8 udp dport 9100 notrack
  }
}

CPU dropped back to where it should have been, a fraction of a percent, basically noise.

The lesson I keep relearning is that "idle" is a userspace word. A machine can be flat-out in the kernel while every process looks asleep, and the only honest way to see it is to sample the CPU directly rather than ask the process list to confess. perf top is the first thing I reach for now whenever a box is busy doing, apparently, nothing.