[SOLVED] Fastest RAID storage for single user with lots of data (>50TB)

[brose]

Hi There, First time novice poster who builds my own sole-use server, looking for some advice from IT pros here about the fastest local storage setup I can create (in terms of Read/Write).

I’m an economist doing research with large data. About 40-50TBs stored locally and file sizes as big as 80GB. I personally built my current Raid 60 setup about 4 years ago (24x3TB drives, 12 drives in each Raid6 striped with Raid0). I’m now looking to build something larger (~100TB) and faster in the next few months. I’m the only user on this machine, but even then the analysis can take hours/days (depending on sample size) primarily due to io speeds. Current speeds are in the range of 800-1200 MB/s. All of my data is backed up online, so I’m strictly looking to build something that has a good balance of having fast read/write speeds, with low risk of downtime due to the Raid degrading.

My question is what is the fastest Raid setup I can get in the $5000-$10,000 range (Maybe higher if there is a significant advantage)?

My current setup:
ASUS X99-DELUXE II LGA 2011-v3
Xeon 1650 V4
LSI 9361-8i + Intel Raid Expander RES3TV360 (Using LSI Megaraid to manage the Array)
256 GB Kingston DDR4 2400 (Some of the statistical software I use runs the analysis in memory)
Norco 4224 case
Windows 10, no separate server OS, just running as if it’s a regular PC with a big local D: drive.

Things I’ve thought doing, (which I don’t even know if possible with LSI Megaraid):
Expand to 48 drives, and then either doing 4 arrays x 12 drives in Raid 60 (Do I call this Raid 60+0+0?), or possibly 8 arrays x 6 drives Raid 50 (Raid 50+0+0+0+0?). Is this too risky? Is there a better alternative? Or should I be thinking of a separate setup completely? Thanks in advance.

Follow up question
Maybe I can ask more specific follow up questions if there are no answers for the above question.

Using 48 drives as a reference point, what are the speed advantages of Raid 10 (2x24) over Raid 60 (4x12)? What about the speed and ease of rebuilding a degraded array?

If I were to do a Raid 10 setup instead of Raid 60, is there enough overhead that would I need a separate Raid controller card, verses using Windows storage spaces?

Thanks in advance.

 

[kanewolf]

Well, you won't be using 256GB of RAM. The motherboard you list only supports 128GB according to the specs -- https://www.asus.com/us/Motherboards/X99-DELUXE-II/specifications/

 

[brose]

Well, you won't be using 256GB of RAM. The motherboard you list only supports 128GB according to the specs -- https://www.asus.com/us/Motherboards/X99-DELUXE-II/specifications/

Thanks for responding, and sorry if I wasn't clear. That is my current setup, which I've been using with 256GB of RAM for about 3-4 years now. I don't want to distract from the main question too much, but the motherboard does support 256GB of ECC RAM.

 

[marko55]

Few considerations here.

1) You've got a backup and you're purely after performance, so I'm not sure I'd sweat using 50 over 60. 50 frees up a few more disks to add more spans to your drive group too. That said, if your gonna start going over 4TB disks, I'd definitely go 60.

2) To increase your write performance, adding more spans to your drive group will help.

3) You've got a fantastic performing raid card. However, double check the user guide to confirm the max number of drives allowed in a single drive group.

4) Storage spaces won't likely provide better performance. I'd stick to the hardware raid.

5) RAID 10 is regarded as great performing. Write performance scales very well and rebuilds are super fast with larger disks (think 10TB) because there's no parity. It's just a disk copy job.

6) IF (and only if) you have a good amount of repetitive I/O consisting of tiny files, you could look to add SSD cache capability to your RAID card.

7) Final answer, for that budget I'd consider a raid 10 consisting of 22 x 10TB SAS enterprise drives from seagate. That'll land you right around 100TB usable storage, should give you around double or better your current disk performance, great redundancy, and super fast rebuild times

 

[brose]

Few considerations here.

1) You've got a backup and you're purely after performance, so I'm not sure I'd sweat using 50 over 60. 50 frees up a few more disks to add more spans to your drive group too. That said, if your gonna start going over 4TB disks, I'd definitely go 60.

2) To increase your write performance, adding more spans to your drive group will help.

3) You've got a fantastic performing raid card. However, double check the user guide to confirm the max number of drives allowed in a single drive group.

4) Storage spaces won't likely provide better performance. I'd stick to the hardware raid.

5) RAID 10 is regarded as great performing. Write performance scales very well and rebuilds are super fast with larger disks (think 10TB) because there's no parity. It's just a disk copy job.

6) IF (and only if) you have a good amount of repetitive I/O consisting of tiny files, you could look to add SSD cache capability to your RAID card.

7) Final answer, for that budget I'd consider a raid 10 consisting of 22 x 10TB SAS enterprise drives from seagate. That'll land you right around 100TB usable storage, should give you around double or better your current disk performance, great redundancy, and super fast rebuild times

Thank you for your answer, its very helpful.
Just a quick followup to help me understand the scaling of RAID 10. Does it scale linearly? Ignoring other limiting factors like cable speed and bandwidth, if I were to use 44 x 10TB drives, would that be approximately twice as fast as 22 x 10TB drives? Would 11 x 10TB drives be half the speed?

 

[marko55]

That is correct.

I eluded to it before: You've got to be careful of the limits on your RAID card though. I just checked the specs on the 9361-8i and actually I gave you an unrealistic scenario previously. Max drives per drive group on a 9361-8i is 32 (which is ok for your use case), BUT max spans per virtual drive is only 8. If I'm not mistaken, every pair of drives in a RAID 10 counts as a span. So you could create a RAID-10 with 16 disks, which would essentially be comprised of 8 x RAID-1s that would be striped across for read/writes. If you used 10TB drives, that's only around 72TB usable. If you used 14TB drives, that would net you the 100TB of usable space you're looking for but you could not expand beyond that (unless you created a 2nd RAID-10 and in turn a 2nd drive in the OS). If you definitely want this to be on drive in the OS, you'll want to go a different route.

So if you want to scale much higher, you can go with 50 or 60. I'd keep the size of your spans smallish which does 2 things: (1) lowers risk and (2) will improve write performance because you'll have more spans in your drive group. If using enterprise class HDDs, and 4TB or lower, I wouldn't hesitate to use 50 and keep each span to 6 disks or less. If you're going higher than 4TB drives, I'd stick with RAID-60 and depending how courageous you are you could do 12 drives per span (I've done it). EVERYONE has very differing opinions on these numbers....for real. You'll hear some people tell you flat out if you use RAID-5 you're out of your mind, and other people have been running production workloads for 5+ years on a RAID-5 comprised of 20+ x 6TB disks, and I'd say that's absolutely insane.... BUT, the one thing to remember is you've got a backup, so you're doing it right. Your personal risk tolerance ultimately decides how you build your RAIDs in my experience.

So, to clarify:

• Each "drive group" is gonna be a RAID array (50 or 60 or 10), and will present to the OS as a single logical drive.
• Max spans per drive group: 8. So, a RAID 10 can have 8 x RAID-1's in it. A RAID-50 can have 8 x RAID 5s in it. And so on.
• Max physical drives per drive group: 32

So, lets say you used 32 x 6TB drives and built a RAID 60. You'd could do 4 spans, consisting of 8 disks each, and you'd end up with 144TB raw storage (after RAID parity). Figure around 91% of that for actual usable storage for the OS: 131TB usable storage. That would give you approximately 4 x write speed of a single RAID-6. That said, write speeds are a funny thing because of write-back caching, which you can force on your adapter whether you have a BBU or not. So, write performance will scream anyway (depending on your workloads).