A monitoring box pinged me about low disk this morning, which was a surprise because it does almost nothing. The culprit, as it so often is, was the journal. journalctl --disk-usage reported just shy of 4GB, which on a 32GB SSD is a meaningful chunk of nothing-very-much.
By default systemd-journald caps itself at 10% of the filesystem, which is fine on a large disk and absurd on a small one. You can reclaim space immediately by time or by size:
journalctl --vacuum-time=7d
journalctl --vacuum-size=200M
But that's a one-off, and the journal will simply grow back. The durable fix is in /etc/systemd/journald.conf. Uncomment and set a hard cap, then restart the service:
[Journal]
SystemMaxUse=200M
systemctl restart systemd-journald
Two hundred megabytes is plenty of history for a box like this, and it'll never ambush me again. The wider lesson, which I keep relearning, is that "uses 10% of disk" is a sensible default on a server and a daft one on the small, cheap machines that make up most of a homelab. Set the cap when you provision the box, not the morning it fills up.