I have run ZFS on data pools for years. Root on ZFS I had always avoided, on the theory that the one filesystem you cannot afford to lose is the one you boot from, and ext4 has never once let me down there. Last weekend I finally did it on a homelab box that I could afford to brick, and I am glad I did. Snapshots of the entire OS, before every upgrade, is the feature I did not know I was missing.
The appeal is concrete. zfs snapshot rpool/ROOT/default@before-upgrade, then apt full-upgrade, and if the new kernel hangs I roll back the whole root in seconds. No more "I'll just back up /etc and hope". The entire system is one transactional unit.
The layout I settled on follows the dataset-per-thing convention that boot environment tooling expects:
rpool/ROOT/default mounted at /
rpool/home /home
rpool/var/log /var/log
bpool/BOOT/default /boot (separate pool, older feature flags)
The separate bpool for /boot is the bit that trips people up. GRUB only understands a subset of ZFS features, so the boot pool gets created with a deliberately conservative feature set. Put /boot on your main rpool with all features enabled and GRUB may refuse to read it after some future zpool upgrade. Keep them apart.
The gotcha that cost me an evening was the initramfs. On first boot the system dropped to a BusyBox prompt with the pool unimported. The cause was that the ZFS root hook was not in the initramfs because I had built it before installing zfs-initramfs, and the ZPOOL_IMPORT_PATH was not finding the pool by its cache file. The fix was dull and total: chroot back in from the live USB, apt install --reinstall zfs-initramfs, regenerate with update-initramfs -c -k all, and make sure /etc/zfs/zpool.cache actually reflected the imported pool. After that it came up clean every time.
One thing I would tell my past self: set ashift=12 at pool creation and never think about it again, set compression=lz4 because it is free and usually a net speed-up, and leave atime off on root. Defaults on the rest. I went in expecting to tune a dozen properties and the honest answer is that the defaults are fine for a root pool. The value is not in the tuning. It is in the snapshots, and in finally trusting that an upgrade is one command away from being undone.
Would I do this on a production server I cannot easily get console access to? Not yet. The initramfs fragility means a botched upgrade can leave you at a recovery prompt rather than a working SSH session, and that is a worse failure mode than a corrupt ext4 you can fsck remotely. But on the homelab, where the console is two metres away, root on ZFS has earned its place.