The disk was filling up despite logrotate clearly running. du on the log directory showed a few sensible megabytes. df insisted gigabytes were gone. That gap, between what the filesystem says is used and what you can actually find, is almost always a deleted file that something still has open.
lsof | grep deleted confirmed it instantly:
appd 4127 app 3w REG 8,1 9663676416 /var/log/app/app.log (deleted)
The app had the old log file open, kept writing to it, and the file lived on as an unreachable inode because the process held a reference. logrotate had dutifully renamed app.log to app.log.1, created a fresh app.log, and then sent SIGHUP so the app would reopen its handle. The app, written by someone who'd never read about that contract, ignored SIGHUP entirely. So it carried on filling a file that no longer had a name, invisibly, forever.
The clean fix is for the app to reopen its log on SIGHUP, but I didn't own the app and wasn't going to get a patch through in time to save the disk. The pragmatic fix was copytruncate in the logrotate config:
/var/log/app/app.log {
daily
rotate 7
compress
copytruncate
missingok
notifempty
}
copytruncate copies the file's contents to the rotated name, then truncates the original in place rather than renaming it. The app's file handle stays valid because the inode never goes anywhere, and it just keeps writing at offset zero. There's a tiny race where lines written between the copy and the truncate can be lost, which is why it's the fallback rather than the default. But for a chatty app that won't honour SIGHUP, losing a handful of log lines once a day beats losing the disk. Restarted the app to free the orphaned inode, switched to copytruncate, and the df/du numbers agreed again.