Ramblings of an aging IT geek
← Ramblings of an aging IT geek
linux

how much swap, and why i stopped arguing about it

After years of cargo-culting swap sizes, I worked out what my homelab machines actually want, and it depends more on the workload than the RAM.

A rack of homelab servers

The swap question is one of those debates that never ends because everyone is answering a slightly different question. "How much swap should I have?" has no single answer, and the people insisting it does are usually generalising from one workload they once tuned. I have finally settled it for my own machines, and the settled answer is: it depends, but in a small number of knowable ways.

Here is where I landed across the homelab.

the boxes with plenty of RAM and no OOM tolerance

My main hypervisor has 64GB and runs VMs that I genuinely do not want killed. For that machine the old "swap equals RAM" rule is nonsense. I am not hibernating it, so I do not need 64GB of swap to dump memory into. What I want is a modest amount of swap as a release valve for cold pages, so that rarely-touched memory gets pushed out and the page cache gets the room instead.

So: 8GB of swap, and crucially vm.swappiness turned down to 10. Not 0. Zero tells the kernel to avoid swap until it's desperate, which on a memory-pressured box means it thrashes the page cache instead and everything feels slow. Ten says "you may evict genuinely cold anonymous pages, but don't go looking for trouble". That single change made the difference between a host that felt sluggish under load and one that didn't.

A server's memory and swap usage graph

the small boxes that must not fall over

The Pi-class machines and the little NUC running DNS and a few utility services are the opposite problem. They have little RAM and no headroom, and the worst outcome is the OOM killer reaping something I actually need at 3am. On those, swap is a genuine safety net, and zram is the right tool.

zram gives you a compressed block device in RAM that acts as swap. You trade a bit of CPU for effective capacity, and because it never touches the SD card or SSD you avoid wearing the storage out. On a 4GB Pi I give it 2GB of zram swap and leave swappiness at the default-ish 60, because here I genuinely want it leaning on swap before it starts killing processes.

# /etc/systemd/zram-generator.conf
[zram0]
zram-size = ram / 2
compression-algorithm = zstd

That's it. The zram-generator package wires up the device and the swap unit on boot. zstd because it compresses well enough and is fast enough that the CPU cost is invisible on these workloads.

the rule i actually use now

The thing I stopped arguing about is the premise. Swap is not "extra RAM" and it is not a sign of failure. It is a place to put memory the kernel has correctly decided it does not need right now. Whether you want a lot or a little is entirely a question of what you are protecting against: thrash, OOM kills, or hibernation.

My rules, written down so I stop relitigating them:

  • Big box, no hibernation, latency-sensitive: small disk swap, swappiness 10.
  • Small box, tight on RAM, must stay up: zram swap, default swappiness, let it lean.
  • Anything that hibernates: swap at least RAM-sized, which on my fleet is none of them.

The measurable win was on the hypervisor. Before the swappiness change, under a backup run that hammered the page cache, interactive SSH would stutter for seconds at a time. After, it doesn't. I didn't add RAM, I didn't add swap, I just told the kernel to value the right kind of memory. That is the whole debate, really. Not how much, but what for.