Ramblings of an aging IT geek
← Ramblings of an aging IT geek
linux

journald ate the disk again, so i bounded it

Capping systemd-journald disk usage with SystemMaxUse and friends so a chatty service can't quietly fill the root partition.

A terminal showing disk usage output

I got paged for a full root partition and, as is tradition, the culprit was /var/log/journal. A service had got chatty, the journal had grown to a few gigabytes, and the default behaviour of "use up to ten percent of the filesystem" turns out to be a lot of room on a box with a big disk and a small root.

The first thing worth knowing is what's actually in there:

journalctl --disk-usage

That tells you the truth in one line. Then you can either vacuum it down right now, by size or by age, or set a permanent bound so it can't happen again. I do both:

journalctl --vacuum-size=500M

For the permanent fix, edit /etc/systemd/journald.conf and set SystemMaxUse=500M. There's also SystemMaxFileSize and MaxRetentionSec if you care about age rather than size. Restart systemd-journald and it'll trim to the new bound on the next rotation.

The wider lesson is that "ten percent of the disk" is a default tuned for a world that no longer matches your partition layout. Anything that defaults to a percentage of free space deserves a hard cap on a server. I now set SystemMaxUse as a matter of course in the base config, and that page hasn't fired since.