I moved a server's root filesystem onto ZFS this week. ZFS on Linux has been at the 0.6.5 line for a while now and the rough edges I remembered from a couple of years ago have largely been sanded off. Root-on-ZFS still isn't a click-next install, but it's no longer the act of faith it once was.
The reason I bothered: I wanted the same thing everyone wants from ZFS. Snapshots, checksumming I can trust, and zfs send for backups that aren't a tarball. The data volumes were already on a pool. Moving root onto it too meant one consistent story for the whole machine rather than ext4 for / and ZFS for everything else.
the shape of it
The pool layout I settled on keeps root in its own dataset so I can snapshot it independently of the rest:
zpool create -o ashift=12 rpool mirror /dev/disk/by-id/ata-... /dev/disk/by-id/ata-...
zfs create -o mountpoint=none rpool/ROOT
zfs create -o mountpoint=/ rpool/ROOT/default
zfs create -o mountpoint=/home rpool/home
Use by-id paths, not /dev/sda. The day a disk reorders on you, future-you will be grateful. I set ashift=12 explicitly because the drives are 4K-sector and ZFS doesn't always guess right; getting that wrong is painful to fix later because it's baked in at pool creation.
Copying the existing system across was an rsync -aHAX from the running root into the new dataset, with the usual exclusions for /proc, /sys, /dev and the like. Nothing surprising there.
the part that bites
The bit that nearly caught me out is the boot chain. Your bootloader and initramfs need to know how to import the pool and mount the root dataset before the real system exists. On this box that meant making sure the ZFS hooks were in the initramfs and setting the bootfs property so the tooling knows which dataset is root:
zpool set bootfs=rpool/ROOT/default rpool
The specific gotcha: the pool has to be importable at boot, and if it was last touched by a running system that didn't export it cleanly, the import can refuse on a different hostid. The fix is to make sure the initramfs imports by pool name with force only where you genuinely understand why, and that the cache file listing your pools actually made it into the initramfs. I rebuilt the initramfs, double-checked the cache file was inside it, and the next boot came up clean.
I will admit I kept the old ext4 root disk physically in the machine, untouched, for the first few reboots. If ZFS root had failed to come up I wanted a five-minute path back to a working system, not a rescue-USB evening. It came up fine every time, but I'd still recommend the safety net. Don't burn the bridge until you've crossed it twice.
was it worth it
Yes, for this machine. The first thing I did once it was stable was zfs snapshot rpool/ROOT/default@clean-install, and knowing that point exists changes how I treat the box. Upgrades feel reversible. The scrub runs weekly and tells me the data is actually intact rather than me hoping it is.
Would I do it on a laptop, or somewhere I value fast boots and minimal fuss over all this? Probably not yet. Root-on-ZFS is still enough extra moving parts that I'd only reach for it where the snapshots and integrity genuinely earn their place. On a server I plan to keep for years, that's an easy call.