The disk filled up despite logrotate doing exactly what I'd configured it to do. df said the partition was full, du on the log directory said the logs were tiny. That gap is the classic tell: a file has been deleted but a process still holds it open, so the space isn't freed until the process lets go.
What happened is the well-worn one. The default rotate behaviour renames the live log out of the way, creates a fresh empty one, and signals the app to reopen its file handle. Most daemons reopen on SIGHUP and pick up the new file. This particular app did not. It cheerfully ignored the signal and carried on writing to the old inode, now named app.log.1, which logrotate would later compress and delete out from under it. The app kept writing to a file with no name, the kernel kept the blocks pinned, and the disk filled with a log nothing could see.
lsof | grep deleted confirmed it: the process holding a (deleted) handle, still growing.
The fix when an app won't reopen on signal is copytruncate:
/var/log/app/*.log {
daily
rotate 7
compress
copytruncate
missingok
}
Instead of renaming, logrotate copies the file then truncates the original in place. The inode never changes, so the app's open handle keeps pointing at the same file, which is now empty again. There's a tiny window between copy and truncate where a line can be lost, so it's not what you'd choose if reopening on SIGHUP were on the table. But for an application that simply won't cooperate, it beats watching the disk fill with logs you can't read. Restart the app, reclaim the space, move on.