Advice on building a ZFS-based file server

[pineconez]

Okay, great.

I saw your post in nolegrl's thread and decided to go with 2TB disks, for the reasons you mentioned and because I don't immediately need 6TB of storage. For the same money I'd be spending on four 3TB or six 2TB disks, I can get four WD RE4s, i.e. enterprise storage. Any experience with those? Else I'm just going to call this a build and go with Barracudas.

 

[sk1939]

If you go with the Green Barracudas, realise that they are 4 platter disks as well. If your going with Seagate, look for the ST2000DM001 which has 2, 1TB platters.

The RE4's aren't really enterprise drives in the truest sense, rather they are standard WD drives with enhanced testing, enhanced firmware, and TLER support (time limited error recovery) which is extraordinarily important if you use hardware RAID so the drives don't fall out of the array due to timeouts. It's not so important with ZFS or software RAID.

 

[pineconez]

Good, I'll probably go with the ST2000DM001s then.

Sorry for bothering, but there are three more questions I'd like to have answered before I commit to this:

First, is there any such thing as a 8i2e card? I've been flexing my Google skills, but so far, nothing has come up. There are 12i4e cards (from LSI, for example), but those are way over budget. I'd like to have the option of adding a simple external box when the case isn't enough. Otherwise, I'll either go with the Intel SASUC8i, LSI 9211-8i or Supermicro AOC-USAS2-L8i. Those're all based on LSI hardware, and appear to work well with ZFS.

The second question is ZFS-related. I'm wondering whether the RAID-5/RAID-6 problems (UER and write hole) transfer to -Z and -Z2, respectively. If they do, I'll most definitely go for a pool of mirror vdevs. If not, I won't, because I have serious doubts I can max out even RAID-Z performance. (Sure, in theory it's possible, but the most I/O-heavy stuff will be copying files from my desktop or to my desktop...and that one doesn't have a RAID. And I'm living alone, so I'm not in the "have to supply BR-quality movies to 5 persons simultaneously" spot. 😉)

Third. I know that with ZFS I can easily change controllers should mine undergo spontaneous combustion or whatever, but what happens if I change the OS drive, or lose it? Essentially, what I'm asking is this: can I pull the OS drive, do a clean install and then access the storage disks without any trouble?

Thanks in advance

Ed.: In favor of the Intel controller, it automatically disables its write cache if no battery backup is installed (that would be a major criterion, otherwise the thing's useless for ZFS as I understand it).

 

[sk1939]

Well, you could try this thing, and connect it to the SASUC8I, which would give you the external ports your looking for.

http://www.google.com/products/catalog?rlz=1C1GPCK_enUS435US435&q=mini+SAS+adapter+card&um=1&ie=UTF-8&tbm=shop&cid=13360347331066941746&sa=X&ei=5MuWT9jUF8Xc6QHTieXPDg&ved=0CHkQ8wIwBA

As to your second question, ZFS does not have the write hole issue that RAID5 does, and as far as UER goes, this article explains it quite nicely:

http://www.zdnet.com/blog/storage/why-raid-5-stops-working-in-2009/162

As a result I would run RAID-Z2/RAID-6 just for redundancy.

To your third question, I really don't know since the OS drive is usually redundant in at least a RAID 1 configuration. I've never had an OS drive as a single, standalone drive before.

 

[pineconez]

Yep, it'd give me external ports, but then I'm down to 4 lanes on the inside. Ah well, I'll decide that when it comes to it.
Btw. - can you connect more than one drive to one of these lanes? Say, plug in one of those port multipliers, then run 2 or 4 disks off of one of the lanes? I'm aware this would screw with drive performance, I'm just curious whether the controller supports it.

Ok, so Z2 is an option. That's good to know. I'll have to do some extensive performance testing myself, and see if I'm comfortable with it. Else it's likely going to be mirrored vdevs, because with 6x 2TB disks there's a breakeven in capacity anyway (and a pool of mirrored vdevs has the virtue of being incredibly easy to expand).

As for the third question, I'm going to ask around in the FreeBSD fora and see what they have to say on it. Anyway, going to catch some sleep now, thanks for your time

 

[sk1939]

Just add an SAS expander. The controller should support it, as it's no different than an SAS expander.

http://www.amazon.com/CHENBRO-MICOM-CK12803-EXPANDER-28PORT/dp/B001CM4E8O

SAS Expander vs. Port Multiplier

http://www.sasexpanders.com/vs/port-multipliers/

That is true.

No problem.