A monitoring box paged me at the worst possible time, not because anything was on fire but because / was at 96% and climbing. No runaway application, no leaked tempfiles. It was /var/log/journal, sitting at 3.8 GB on a host that does almost nothing except scrape metrics. The journal had simply been allowed to grow until the filesystem said no.
By default systemd-journald will use up to 10% of the filesystem the journal lives on, capped at 4 GB, and it only frees space when it bumps into that ceiling. On a small root partition 10% is plenty to make you uncomfortable, and the default keeps logs effectively forever until then. Fine on a fat server, less fine on a 20 GB VM.
First, see what you are actually dealing with:
journalctl --disk-usage
# Archived and active journals take up 3.8G in the file system.
The fix is two settings in /etc/systemd/journald.conf. I want a hard size cap and a time-based retention window, whichever bites first:
[Journal]
SystemMaxUse=500M
MaxRetentionSec=2week
SystemMaxUse is the total the persistent journal may occupy. MaxRetentionSec discards entries older than the window regardless of size, which matters when a host is quiet for weeks and you would otherwise keep boot logs from the spring. There is also SystemKeepFree, which reserves headroom on the filesystem; the journal honours whichever constraint is stricter.
Apply it and reclaim the space immediately rather than waiting for the next rotation:
systemctl restart systemd-journald
journalctl --vacuum-size=500M
# or, if you prefer to vacuum by age:
journalctl --vacuum-time=2weeks
The vacuum commands are also handy as a one-off when you are mid-incident and just need the partition to breathe. They operate on archived journals only, so they will not touch the currently active file, which is exactly what you want.
One thing worth knowing: if /var/log/journal does not exist, journald keeps logs in /run, which is tmpfs, so they vanish on reboot and never threaten the disk. Some minimal images ship that way deliberately. The moment you mkdir /var/log/journal you have opted into persistence, and you have also opted into managing its size. I had forgotten I did exactly that on this host months ago when I wanted boot logs to survive a crash.
Since then I bake those two lines into the base image. Five hundred megabytes of logs is more than enough to debug anything I am likely to debug, and the alert that woke me has not come back.