My NAS ran out of space the week before Christmas, which is exactly when you'd least want to be elbow-deep in a storage migration. So naturally that's when I did it. This is the story of swapping every disk in a running pool for a bigger one, without taking the array down and, more importantly, without losing anything. ZFS made it possible. My nerves did not entirely survive.
The starting position
The box was running FreeNAS, four 3TB drives in a single RAID-Z2 vdev. That's two drives of redundancy, which is the right amount of paranoia for irreplaceable data: family photos, a decade of documents, the media library that justifies the whole thing to the household. The pool was sitting at about 88% full, and ZFS gets unhappy and slow well before 100%, so this wasn't a want, it was a need.
The plan was the classic one for growing a RAID-Z pool without rebuilding it from scratch: replace each disk with a larger one, one at a time, let the pool resilver after each swap, and once the last of the four is done the pool quietly grows to use the new capacity. Four 3TB drives become four 8TB drives, and the pool roughly triples without ever going offline.
It works because of autoexpand. With it on, once every member of a vdev is bigger, ZFS expands to fill the new space automatically. Mine was off by default, which is a detail I'm very glad I checked before I started rather than after.
zpool set autoexpand=on tank
zpool get autoexpand tank
The shuffle itself
The actual procedure per disk is calm right up until it isn't. You offline the old drive, physically pull it, put the new one in the same bay, and tell ZFS to replace it.
zpool offline tank gptid/<old-disk-id>
# physically swap the drive in that bay
zpool replace tank gptid/<old-disk-id> gptid/<new-disk-id>
zpool status tank
Then you wait. And this is the part the guides gloss over: the resilver on the first disk took the better part of a day. The pool was full, the drives are spinning rust, and resilvering is essentially a full read of all the data to reconstruct the new disk's share of it. zpool status ticking along at a few percent an hour is not a relaxing thing to watch when you're already a member down on your redundancy.
The thing that focused the mind: during each resilver, the pool is degraded. RAID-Z2 means I could survive losing one more drive during that window, but only one. So for the better part of a day, four separate times, I was running with less safety margin than I'm comfortable with. That's the real cost of the one-at-a-time approach, and it's why you do not start this without a current, verified, off-the-box backup. I had one. I checked it twice. I cannot stress this enough: the redundancy is for hardware faults, not for the mistakes you make at midnight.
The scare
Disk three is where it got interesting. Partway through its resilver one of the other old drives, one I hadn't swapped yet, started throwing read errors in zpool status. For about ten minutes I genuinely thought I was going to find out whether my backup restore actually worked, in anger, on Christmas week.
It held. The errors were correctable, ZFS healed them from parity, the resilver finished, and the drive that scared me got swapped out next as planned with no further drama. But it was a useful reminder that old drives don't fail politely on a schedule. They fail when you're already stressing them, which is precisely during a resilver. If you're doing this, expect at least one moment where you question your life choices.
Landing it
After the fourth disk finished, the moment of truth:
zpool status tank # all four new disks, no errors
zpool list tank # capacity has grown
zpool scrub tank # then a full scrub for peace of mind
The pool expanded on its own thanks to autoexpand, exactly as advertised. I kicked off a scrub immediately, which read the whole thing back and confirmed everything was consistent, and went to bed at a genuinely unreasonable hour feeling quietly triumphant.
What I'd do differently
A few honest lessons:
- Don't do this at 88% full. The resilvers are slower and the margin for error is thinner. Migrate when you've got headroom, not when you're desperate.
- Burn-in the new drives before they go anywhere near the pool. I ran a long SMART test on each beforehand and it caught nothing, but it's cheap insurance against putting a dead-on-arrival disk into a degraded array.
- Have the backup, and test the restore, before you pull the first drive. Not as a formality. As the actual plan B you might have to use.
- Accept that it'll take days, not hours. Four resilvers plus a scrub on a full pool of spinning disks is a week of background grinding. Be patient and don't poke it.
ZFS earned every bit of its reputation here. Replacing the entire physical substrate of a storage pool while it stays online, and having it heal itself when an old disk wobbled mid-flight, is genuinely the sort of thing that makes the homelab worth the bother. I now have triple the space and a renewed, slightly nervous respect for the phrase "degraded pool". Next time, though, I'm migrating before I'm out of space, not after. Lesson learned.