I'd run ZFS on data pools for years and kept root on plain ext4, because root-on-ZFS on Linux always had a faint whiff of "here be dragons" about it. This week I finally moved a machine's root filesystem onto ZFS, and the thing that pushed me over the edge wasn't checksums or compression, lovely as those are. It was the boot environment. The ability to snapshot the entire OS before a risky upgrade and roll the whole thing back if it goes wrong, in seconds, is the feature I'd been quietly wanting for a decade.
The reason root-on-ZFS is fiddly on Linux comes down to one awkward fact: the licence. ZFS can't be merged into the kernel, so it ships as an out-of-tree module via ZFS on Linux, and your bootloader has to be able to read a ZFS pool before that module is loaded. That chicken-and-egg between "the filesystem is a module" and "the bootloader needs the filesystem to find the module" is the source of most of the pain. It's all solved, the distro guides walk you through it, but it's why this was never quite as casual as mkfs.
The mechanics, once you accept the bootloader dance, are pleasant. You create a pool, you carve it into datasets that map to parts of the OS, and you let the boot environment tooling handle the snapshot-and-promote shuffle.
zfs list -o name,used,mountpoint
NAME USED MOUNTPOINT
rpool/ROOT 8.21G none
rpool/ROOT/default 8.20G /
rpool/home 3.44G /home
The split that matters is keeping the OS (rpool/ROOT/default) separate from the things you don't want rolled back with it, like /home. That way, when you snapshot the boot environment before a dist-upgrade, you capture the system without dragging the user data into the rollback. Roll back a bad upgrade and your documents are untouched, because they were never part of the snapshot.
The payoff arrived almost immediately, which is rare. A kernel update on this box left it not booting cleanly. Old me would have reached for a rescue USB and a slow evening of swearing. Instead I picked the previous boot environment from the bootloader, came up on the old kernel, fixed it at my leisure, and rolled forward again. The snapshot was free, the rollback was a single boot menu selection, and the whole ordeal was about ten minutes instead of an evening.
It's not all upside. ZFS likes RAM, the ARC will happily eat memory you thought was free (it gives it back under pressure, but it alarms the unprepared), and the out-of-tree module means a kernel upgrade can occasionally land before a matching ZFS build does, which is its own special way to not boot. You want your snapshots working before that happens, not after.
Would I do it again? On anything I care about, yes. The honest summary is that root-on-ZFS doesn't make a machine faster or simpler. It makes it recoverable, and it turns "I'm about to upgrade and I'm slightly nervous" into "I'll snapshot first and stop worrying". For a box you maintain rather than redeploy, that trade is well worth the bootloader faff.