The pool layout I chose three years ago no longer fit the data, and the only way to change a ZFS vdev's geometry is to destroy it and rebuild. So this weekend was the great disk shuffle: moving everything off, reshaping the pool from a stack of mirrors into a single RAIDZ2 vdev, and moving it all back. With one storage server and no spare copy of several terabytes, this is the kind of job where you read your own notes twice before typing anything.
The timing was convenient. iXsystems renamed FreeNAS to TrueNAS CORE earlier this year, and the 12.0 release is the one that carries the new name properly. I'd been putting off the upgrade, and "I'm rebuilding the pool anyway" was as good an excuse as I was going to get. So this was a clean install of TrueNAS CORE 12.0 rather than an in-place upgrade, which I prefer for anything I'm going to lean on for years.
The plan, written out before I touched a single screw:
- Build a temporary pool on a set of older drives I'd kept for exactly this.
zfs sendthe datasets across, snapshots and all, rather than copying files. This preserves properties and is restartable if it dies halfway.- Destroy the old pool, fit the new disks, create the RAIDZ2 vdev.
zfs sendeverything back, then verify before trusting it.
The send/receive is the part worth getting right. Doing it with snapshots means it's atomic per dataset and you can resume an interrupted transfer:
zfs snapshot -r tank@migrate
zfs send -R tank@migrate | zfs receive -F temp/tank
The -R carries the whole tree, child datasets, properties, the lot. On the way back it's the same idea in reverse. I piped it through mbuffer to smooth out the throughput, because raw send/receive across a busy box stutters in a way that wastes hours.
Two things bit me, both my own fault. First, I hadn't accounted for how much slower RAIDZ2 resilvering would be on nearly-full disks, so the final verification ran well into Sunday evening. Second, I'd forgotten that a couple of jails had local data outside the datasets I was snapshotting, and nearly destroyed the old pool before noticing. Nearly. The lesson, again, is that the dangerous bit of a migration is never the bit you planned for, it's the small thing sitting just outside your mental model.
It's done. RAIDZ2 gives me two-disk redundancy with far better capacity than the mirror stack, TrueNAS CORE 12.0 is running happily, and the data verified clean. I'd not want to do it every weekend, but for a box that's going to hold my files for the next several years, an afternoon of careful zfs send was cheap insurance. Now to put the borrowed disks back before anyone notices they're warm.