One thing that's been unclear to me with ZFS: How should I design a ZFS-based storage system for extensibility? From what I've read, it seems like you can't add devices to a pool once it's been created. I'd love to create a small-ish NAS right now to store, say, the top 10% of my blu-ray collection, and then incrementally upgrade it over time as I get funds.
The easiest way to do this is to use mirror vdevs, that way you can add disks two at a time. Of course, you miss out on RAIDZ functionality that way, but that's the simplest setup.
I've gone ahead and actually created a new array using btrfs raid10 instead (with an eye towards a RAIDZ2/3 like configuration whenever the btrfs guys get around to fixing it); btrfs allows for online reshaping of the mirroring configuration, so it doesn't have the limitations of ZFS in this instance.
I'm still not sure what setup I have. I really should have written down every decision I've made while building my NAS (I think I've now found articles that disagree with every single choice I made)
$ zpool status
pool: BoxODisks
state: ONLINE
scan: scrub repaired 0 in 9h20m with 0 errors on Sun Jul 31 01:31:31 2016
config:
NAME STATE READ WRITE CKSUM
BoxODisks ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
gptid/18059e22-b6a4-11e5-9cca-0cc47a6bbf34 ONLINE 0 0 0
gptid/194649b1-b6a4-11e5-9cca-0cc47a6bbf34 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
gptid/1a86b3cc-b6a4-11e5-9cca-0cc47a6bbf34 ONLINE 0 0 0
gptid/1bcd3ad6-b6a4-11e5-9cca-0cc47a6bbf34 ONLINE 0 0 0
cache
diskid/DISK-S24ZNWAG903847Lp1 ONLINE 0 0 0
I think I have mirrored vdevs.
What does that mean for replacing failed disk/s? What does that mean when I want to upgrade to 8TB drives? The biggest "mistake" I made was buying a very expensive 4 bay enclosure - 4 drives is not enough.
When you want to upgrade, you can offline and replace each disk, then resilver, in turn. If the pool is set to autoexpand, you'll eventually get a total capacity of 16TB. You can do this for all the disks, or for each mirror separately in two groups of two.
Unless you want to reshape your array. Then md-raid is superior. Btrfs's raid implementation has had some pretty bad issues recently and has been declared "experimental" as a result.
Apologies, I only saw the recent mailing list post which basically said "we're removing the ability to even compile support for raid56". I didn't mean to say that btrfs was overly-zealous in their trust of new code (I personally am super excited for the future of btrfs).
Yeah, I saw that before, but that isn't quite what I wanted, if I'm reading it right. That tells me how to replace all the drives in my array with new ones; I'd prefer to (for instance) start with 4x 4TB drives, and add new ones as my storage needs build up.
You can expand a pool one RAID array at a time, but growing one drive at a time isn't really possible except by not using RAID; ZFS doesn't have a true equivalent to the BTRFS rebalance operation.
