how i moved every disk in my nas without losing a byte

My NAS had become a museum of disks. Two 4TB drives bought in 2019, two 6TB drives bought when one of the 4s died, and a single 8TB that I added in a panic during a near-full event and then never tidied up. The pool worked, it was healthy, but it was a single RAIDZ2 vdev built from the smallest common denominator, which meant I was paying for terabytes that ZFS politely refused to use. I had five 12TB drives sat in their boxes. This is the story of getting from one to the other without taking the array offline and without doing anything I would regret at 2am.

The short version, because it is the whole point: you can grow a ZFS RAIDZ vdev by replacing every disk in it one at a time, waiting for each resilver to complete, and once the last small disk is gone the pool autoexpands to the new size. It is slow, it is boring, and it is the safest way to do this that exists. I did it the boring way on purpose.

Why not just rebuild the pool

The tempting alternative is to back everything up, destroy the pool, create a fresh one from the five new drives, and restore. It is faster on paper, and if you have somewhere to put the data it is arguably cleaner. I did not do it, for two reasons. First, my offsite backup is real but it is on the end of a domestic upload link, and restoring 14TB over it would take the better part of a week during which I would have no local copy and no redundancy. Second, and more honestly, the destroy-and-recreate path has a window where the only copy of my data is the backup, and I have a deep, hard-won distrust of any plan whose happy path depends on a restore I have not rehearsed that morning.

The disk-at-a-time replace keeps full RAIDZ2 redundancy throughout, minus one drive's worth during each resilver. At no point is the data in a worse state than "could survive one more failure". That margin is the entire reason I sleep at night.

The actual procedure

TrueNAS SCALE makes this less hair-raising than the command line, but I like to know what it is doing underneath, so here is the shape of it. For each old disk:

1. Physically install the new drive if you have a spare bay, or note which old drive you are about to pull if you do not.
2. In the pool's status view, choose the old member and replace it with the new one. SCALE wires up zpool replace for you.
3. Wait. Watch the resilver. Do not start the next one until this one is done and the pool is back to a clean ONLINE state with no errors.

If you are doing it on the command line, it is exactly what you would guess:

# identify the old member's gptid and the new drive
zpool status tank

# kick off the in-place replacement
zpool replace tank <old-gptid> /dev/disk/by-id/<new-drive>

# watch it, because you will anyway
zpool status tank 1

The bit nobody tells you up front is how long a resilver actually takes on near-full spinning disks. Mine were running at roughly 120 to 150 MB/s sustained, slower than the raw drive speed because resilvering reads scattered across the whole pool rather than streaming nicely. Each 6TB member took the better part of eleven hours. Five replacements, back to back, with me refusing to overlap them, turned into the better part of a working week of elapsed time. The array was online and serving the whole while, which is the trade you are making: maximum safety, minimum speed.

The bit that bites people: autoexpand

Here is the gotcha that catches everyone the first time. You replace all five disks, the last resilver finishes, and the pool is still showing the old size. Nothing grew. The fix is one property:

zpool set autoexpand=on tank
zpool online -e tank <gptid>   # nudge a member if it is stubborn

In my case SCALE had autoexpand on already and the pool grew the moment the final resilver completed, but I have been bitten by this on a hand-rolled pool before, sat there confused, staring at a pool that was full of big disks and still the size of the small ones. If you take one thing from this, take that: the expansion does not happen until the smallest disk in the vdev has been replaced, because RAIDZ sizes to its smallest member. Four big disks and one small one buys you nothing.

What I would do differently

Two things. First, I would have labelled the physical drive bays years ago. I spent a genuinely embarrassing amount of this exercise cross-referencing serial numbers against zpool status with a torch, terrified of pulling the wrong drive mid-resilver, which is exactly the moment you least want to lose a second one. A label maker is the cheapest insurance in the rack.

Second, I would have scrubbed the pool before I started, not after. I scrubbed at the end and it came back clean, which was a relief, but if there had been latent corruption sitting on those old drives I would have far rather known about it before I spent a week feeding data through resilvers. Scrub first. Know the ground is solid before you start moving the furniture.

The pool is now five 12TB drives in RAIDZ2, healthy, scrubbed, with room to be careless again for a few years. It cost me a week of patience and nothing else. The data never moved, never went offline, and never lived anywhere but on disks that could lose two of their number and shrug. That is the whole reason to do it the slow way.