A service was burning more CPU than it had any right to, and I had a list of suspects. The JSON encoding. The database driver. That one regex I'd been meaning to rewrite. All of them wrong.
I grabbed a profile with perf record -F 99 -g, folded it through Brendan Gregg's scripts, and opened the SVG. The widest plateau, by a comfortable margin, was in our structured logging. We were building a fully formatted log line, fields, timestamps, the lot, for every request at debug level, and then discarding it because the configured level was info. The work happened before the level check. So we paid for log lines nobody would ever see.
The fix was embarrassing in its smallness: guard the expensive formatting behind the level test, so the fields only get assembled if we're actually going to write them. CPU on that service dropped by about a third, and the latency tail came in with it.
I've since stopped guessing. A flamegraph takes five minutes to produce and it has never once pointed where I expected. The hot path is always somewhere boring, and it's usually something I wrote whilst thinking about something else.