I have run ZFS on data pools for years. The NAS, the backup target, anything where I cared about bit rot and wanted cheap snapshots. What I had always shied away from was putting the root filesystem itself on ZFS, because the failure mode of getting it wrong is a box that does not boot, and I like my boxes to boot.
This month I finally did it on a workstation rebuild, and the thing that pushed me over was boot environments. The ability to snapshot the entire root before an upgrade, run the upgrade, and roll the whole thing back to a known-good state in seconds if it goes wrong. Not the home directory, the whole operating system. That is worth the faff, and there is some faff.
the shape of the setup
The layout I settled on is a single pool, call it rpool, with separate datasets for the bits that should and should not roll back together. Root and the system live in one boot environment dataset. Home lives in its own dataset that is deliberately not part of the boot environment, so that rolling back the OS does not undo this morning's work.
rpool/ROOT (container)
rpool/ROOT/default (the live boot environment, mounted at /)
rpool/home (mounted at /home, outside the BE)
rpool/var/log (so logs survive a rollback)
The principle is simple once it clicks. Anything you want to revert when an upgrade breaks goes inside the boot environment. Anything you want to keep regardless lives outside it. Logs are an interesting case: you almost always want to keep them, because after a bad upgrade the logs are the evidence, so I pull /var/log out of the rollback path on purpose.
the bootloader is the sharp edge
GRUB and ZFS have an awkward relationship. GRUB's ZFS support lags the pool features, so if you create a pool with everything switched on, GRUB may simply refuse to read it, and you will discover this at the least convenient moment. The defensive move is a small separate pool just for boot, with conservative feature flags, or to be disciplined about which features you enable on the root pool.
I went the separate-boot-pool route. A modest bpool with a deliberately reduced feature set holds the kernels and initramfs, and the larger rpool holds everything else with all the modern features on. It is slightly more to reason about, but it means I can use compression and the newer goodies on root without playing chicken with the bootloader.
The other thing that bites people is the initramfs. It needs the ZFS modules baked in and the right hook to import the pool and mount the correct boot environment before the system proper starts. Get this wrong and you drop to an emergency shell staring at a pool that exists but is not mounted. Test it before you trust it. Reboot a few times on purpose, while you still remember how it is wired, rather than finding out in three months when you have forgotten everything.
what it actually buys you
The snapshots are the headline and they earn it. Before any system upgrade I take a snapshot of the boot environment, and the cost is effectively nothing because ZFS snapshots are copy-on-write. If the upgrade is fine, I forget the snapshot exists. If it is not, I roll back and reboot, and the machine is exactly as it was an hour ago, kernel and all.
Compression is the quiet win nobody mentions enough. With lz4 on, most of my root filesystem compresses for free, the CPU cost is negligible, and on modern disks the reduced read volume can actually make things faster rather than slower. It feels like cheating.
Then there is zfs send. I can stream a snapshot of the entire root to another pool, or another machine, as a single consistent image. That is a backup of the operating system that I can restore as a unit, not a tar of files and a prayer about ordering.
would I recommend it
For a laptop you reinstall on a whim, probably not. The boot complexity is real and a fresh install is cheap. For a workstation or a server you actually maintain over years, where an upgrade going sideways costs you a day, yes, without hesitation. The first setup is an afternoon of reading and a couple of deliberate test reboots. After that it fades into the background and only reappears, gloriously, on the day an upgrade tries to ruin your week and you undo it in thirty seconds instead.
The one piece of advice I would press on anyone trying this: write down your exact pool layout and your initramfs configuration somewhere outside the machine. The day you need it is the day the machine will not boot to show it to you.