For years, "why is this box slow" ended at the same wall. I could see the symptom in top or iostat, I could guess at the cause, and then I was reduced to strace on one process at a time, which slows the thing down so much it changes the answer. eBPF, with bpftrace on top of it, finally lets me ask the kernel directly and get a reply without rebuilding or rebooting anything.
The bit that sold me was a one-liner. Which syscalls is this machine actually making, right now, ranked?
bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[probe] = count(); }'
Let it run a few seconds, hit Ctrl-C, and you get a tidy tally. No recompiling, no instrumenting the application, no measurable slowdown. It is reading tracepoints the kernel already exposes and aggregating in kernel space, so the cost is genuinely small.
The one that made me sit up was per-syscall latency as a histogram, on a live production-shaped box, with the slow tail right there in front of me instead of inferred from averages. I have spent whole afternoons trying to reconstruct that picture from logs and metrics, and here it was in two lines.
You need a recent-ish kernel for the good stuff and root to attach the probes, and there is a real learning curve once you go past the canned one-liners. But the shape of it has changed how I think about a slow machine. The kernel was always seeing all of this. I just never had a way to ask. Now I do.