Disk filled up on a box that should never have filled up. df said 100%, du pointed at /var/log/theapp/, and there sat app.log.1 at eleven gigabytes whilst app.log, the file logrotate had just created, was zero bytes and clearly nobody's friend.
This is the classic copytruncate-vs-create trap, except it wasn't even that subtle. logrotate was configured with create, which renames app.log to app.log.1 and makes a fresh app.log. Perfectly sensible, as long as the application reopens its log file when told to. The way you tell it is the postrotate script, which sends a signal, usually SIGHUP. And this particular application, a vendor binary with no source and a support contract that exists mainly to disappoint, ignores SIGHUP entirely. So after rotation the process still held an open file descriptor pointing at the renamed inode, and kept happily writing to app.log.1 forever. New file empty, old file unbounded, disk doomed.
You can confirm it in seconds with lsof:
lsof -p $(pidof theapp) | grep app.log
theapp 4821 app 7w REG 252,1 11811160064 ... /var/log/theapp/app.log.1 (deleted)
There's your smoking gun: an open handle to the rotated, now mis-named file.
The honest fix is copytruncate, which copies the file then truncates the original in place, so the application's descriptor keeps working because the inode never changes. It has a small race window where lines logged mid-copy can be lost, but for a vendor binary that won't take a signal it's the pragmatic answer. I added it, restarted the process once to free the orphaned eleven gigs, and moved on. Some battles you win by refusing to fight the binary on its own terms.