Ramblings of an aging IT geek
← Ramblings of an aging IT geek
linux

journald ate the disk, here is the one line that stops it

A root partition slowly filling with systemd journal logs, and the SystemMaxUse setting that caps it for good.

A Linux terminal showing disk usage

Small box, small root partition, and one morning df telling me it was 94% full with nothing obviously to blame. The culprit was /var/log/journal, sitting at almost three gigabytes of systemd journal, quietly growing since install because nobody had told it not to.

You can see what it's using with journalctl --disk-usage. To reclaim space right now there are two ways: cap by size or by age.

journalctl --vacuum-size=200M
journalctl --vacuum-time=2weeks

Either is a one-off, though, and the journal will just grow back. The actual fix is in /etc/systemd/journald.conf. Uncomment and set SystemMaxUse:

[Journal]
SystemMaxUse=200M

Then systemctl restart systemd-journald and it enforces the cap from then on, rotating out the oldest entries once it hits the limit. On a server with a generous disk the default (10% of the filesystem) is fine and you'll never notice. On anything small, set the cap explicitly. The default isn't wrong, it just assumes you have room to spare, and on a tiny root partition you don't. Two hundred megabytes of recent logs is plenty for working out what broke last night.