The symptom was a disk filling up overnight despite logrotate doing exactly what it was told. The log file was rotated, compressed, the old one deleted, and the box still ran out of space by morning. df said the partition was full. du on the log directory said it was nearly empty. That gap is the whole story.
The app had an open file descriptor on the log file. When logrotate renamed app.log to app.log.1 and then created a fresh app.log, the daemon carried on writing to the same inode it had open all along, which was now the renamed file. The space wasn't reclaimed because a process still held it open, so the kernel kept the blocks around even though the path was gone. Classic deleted-but-open-file behaviour. lsof | grep deleted shows it plainly.
$ lsof -nP | grep '(deleted)'
appd 2451 app 3w REG 8,1 41203847168 131074 /var/log/app/app.log.1 (deleted)
There it is: 41 GB held by a file with no name. Restarting the daemon would release it instantly, and indeed that's what the on-call person had been doing every morning without understanding why.
The usual fix is the postrotate stanza, which sends the daemon a signal telling it to reopen its log files:
/var/log/app/*.log {
daily
rotate 7
compress
missingok
notifempty
postrotate
/bin/kill -HUP $(cat /var/run/appd.pid)
endscript
}
Except this particular app ignored SIGHUP entirely. It had no handler, so the default disposition applied, which for HUP is to terminate. Sending HUP either did nothing useful or killed the daemon outright depending on how it was started. The author had simply never wired up log reopening, which is more common than you'd hope in small in-house services.
You have three honest options when an app won't reopen its logs. Patch the app to handle a signal and call freopen on its log path, which is the right answer if you own the code. Use copytruncate in logrotate, which copies the file then truncates the original in place so the descriptor stays valid, at the cost of a small race where lines written during the copy are lost. Or stop writing to a file at all and log to stdout, letting systemd's journal or a supervisor handle rotation, which is where I'd push any new service today.
We went with copytruncate as the stopgap because we couldn't ship an app change that week, and the lost-line race didn't matter for what was, frankly, debug noise. The real lesson is older than logrotate: a rename doesn't reach into a running process and change where its open descriptors point. The kernel doesn't care what you called the file. It cares what's still holding it open.