I have run ZFS on data pools for years. The root filesystem I had always left on ext4, on the grounds that booting is the one place where you genuinely do not want surprises. Last weekend I finally moved root onto ZFS as well, and I want to write down the bits that mattered before I forget them.
The motivation was boring and correct: I wanted boot-environment style snapshots. Snapshot before an apt upgrade, roll back if it goes wrong, get on with my life. You can approximate this with LVM, but ZFS snapshots are cheap, instant, and I already trust the tooling.
The layout that actually worked
The thing that trips people up is that GRUB's ZFS support is fussy about pool features. The standard advice, which I followed, is to keep a small separate boot pool with a conservative feature set, and a normal root pool with everything enabled.
bpool 1G feature set GRUB understands, /boot
rpool rest of disk, encrypted, everything on
Then datasets rather than one big blob, so I can snapshot and set properties independently:
rpool/ROOT/debian mountpoint=/
rpool/home mountpoint=/home
rpool/var/log mountpoint=/var/log
rpool/var/lib/docker mountpoint=/var/lib/docker
Splitting /var/lib/docker out matters more than it looks. Docker churns through layers, and you do not want a root snapshot dragging gigabytes of dead image layers along with it. Give it its own dataset and let it rot in peace.
The encryption decision
I used native ZFS encryption rather than LUKS underneath. The argument for LUKS is that it is older and more boring, which are both virtues at the bottom of a storage stack. The argument for native encryption is that it travels with zfs send, so my backups are encrypted at rest without re-wrapping. I went native, with a key file on a USB stick and a passphrase fallback, and accepted that I am now slightly more exposed to ZFS encryption bugs than I would otherwise be. That is a real tradeoff, not a free lunch, and if your threat model is serious you should think harder than I did.
Did it earn its keep
Yes, within a week, which is faster than I expected. A kernel update pulled in a DKMS rebuild that didn't, and the box came up without a working module for the very filesystem it was sitting on. With ext4 that's a rescue-USB evening. Instead I rolled the root dataset back to the pre-upgrade snapshot from the GRUB menu, booted the old environment, pinned the kernel, and went to bed.
zfs rollback rpool/ROOT/debian@before-upgrade
The honest caveat: this is more moving parts than ext4, and ZFS-on-root will occasionally remind you of that at the worst moment, usually involving an initramfs that has forgotten how to import the pool. Keep a rescue USB with the matching ZFS version and know how to import and mount by hand. With that safety net in place, I'd do it again, and on the next rebuild I will.