Ramblings of an aging IT geek
← Ramblings of an aging IT geek
homelab

truenas and the great disk shuffle

Growing a ZFS pool on TrueNAS by replacing disks one at a time, the resilver-and-wait dance, and the SMART warning that nearly ruined it.

A server rack with drive bays

The pool was full. Not "getting full", full, the kind where TrueNAS starts nagging and snapshots stop being free. I had a mirror vdev of two 4TB drives and a plan that anyone who's run ZFS will recognise as both correct and tedious: replace each disk with a bigger one, let it resilver, and once both are swapped the pool grows to the new size on its own.

The plan is sound. The execution is an exercise in patience and trusting a progress percentage.

one disk at a time

ZFS won't expand a mirror until every disk in the vdev is bigger. So the dance is:

zpool offline tank <old-disk>
# physically swap the drive
zpool replace tank <old-disk> <new-disk>
# wait for resilver
zpool status tank

Resilvering the first 4TB drive onto an 8TB took most of a day. During that window you are running on reduced redundancy, which is the part nobody enjoys, because if the other disk picks that moment to die you're restoring from backup. I do have backup. I would still rather not.

A homelab cabinet with disks and cabling

the smart warning that nearly ruined it

Halfway through resilvering the second disk, the remaining old drive threw a SMART warning: a handful of reallocated sectors, climbing. Not failed, but unhappy, and unhappy at the precise moment it was the only copy of half my data. I stopped breathing for a second, checked the resilver had passed the worst of it, and let it finish before I touched anything else.

It held. Once the second replacement completed, autoexpand was already on, so the pool jumped from 4TB to 8TB usable without any further prompting:

zpool set autoexpand=on tank   # set this before you start, not after

That last point is the one I'd tattoo on the inside of my eyelids. Set autoexpand=on before the shuffle. If you forget, the disks are bigger but the pool stubbornly stays the old size, and you end up running an online expand per device to convince it. Not hard, just one more thing in a process that already asks you to stare at a percentage for two days.

Final tally: a full weekend, two new drives, one frightening SMART warning, and a pool that's now comfortably half empty. The old 4TB drives are going into a cold spares box, because the next great disk shuffle is only ever a few snapshots away.