Ramblings of an aging IT geek
← Ramblings of an aging IT geek
linux

moving root onto zfs without losing my nerve

Notes from migrating a Debian root filesystem onto ZFS, including the boot pool, datasets, and the snapshot that paid for itself within a week.

A terminal glowing in a dark room

I have run ZFS on data pools for years. The root filesystem I had always left on ext4, on the grounds that booting is the one place where you genuinely do not want surprises. Last weekend I finally moved root onto ZFS as well, and I want to write down the bits that mattered before I forget them.

The motivation was boring and correct: I wanted boot-environment style snapshots. Snapshot before an apt upgrade, roll back if it goes wrong, get on with my life. You can approximate this with LVM, but ZFS snapshots are cheap, instant, and I already trust the tooling.

The layout that actually worked

The thing that trips people up is that GRUB's ZFS support is fussy about pool features. The standard advice, which I followed, is to keep a small separate boot pool with a conservative feature set, and a normal root pool with everything enabled.

bpool   1G   feature set GRUB understands, /boot
rpool   rest of disk, encrypted, everything on

Then datasets rather than one big blob, so I can snapshot and set properties independently:

rpool/ROOT/debian      mountpoint=/
rpool/home             mountpoint=/home
rpool/var/log          mountpoint=/var/log
rpool/var/lib/docker   mountpoint=/var/lib/docker

Splitting /var/lib/docker out matters more than it looks. Docker churns through layers, and you do not want a root snapshot dragging gigabytes of dead image layers along with it. Give it its own dataset and let it rot in peace.

A server in a rack, blinking quietly

The encryption decision

I used native ZFS encryption rather than LUKS underneath. The argument for LUKS is that it is older and more boring, which are both virtues at the bottom of a storage stack. The argument for native encryption is that it travels with zfs send, so my backups are encrypted at rest without re-wrapping. I went native, with a key file on a USB stick and a passphrase fallback, and accepted that I am now slightly more exposed to ZFS encryption bugs than I would otherwise be. That is a real tradeoff, not a free lunch, and if your threat model is serious you should think harder than I did.

Did it earn its keep

Yes, within a week, which is faster than I expected. A kernel update pulled in a DKMS rebuild that didn't, and the box came up without a working module for the very filesystem it was sitting on. With ext4 that's a rescue-USB evening. Instead I rolled the root dataset back to the pre-upgrade snapshot from the GRUB menu, booted the old environment, pinned the kernel, and went to bed.

zfs rollback rpool/ROOT/debian@before-upgrade

The honest caveat: this is more moving parts than ext4, and ZFS-on-root will occasionally remind you of that at the worst moment, usually involving an initramfs that has forgotten how to import the pool. Keep a rescue USB with the matching ZFS version and know how to import and mount by hand. With that safety net in place, I'd do it again, and on the next rebuild I will.