[SOLVED] 500+ TB Storage Server

[marko55]

Alright guys and gals, I've deployed a few large scale storage solutions in the past, but looking for ideas on a new solution. I've got someone that needs a storage server that could scale past 500TB, which would normally not be a big deal except they're asking of all the storage can be one one logical drive in Windows. I'm pondering possible ways to do this and here's where I'm at:

1) HBA and JBOD chassis, presenting to Storage Spaces in Server 2016 and one large ReFS volume. I've seen this done before and it's supported, and technically possible, but I've also seen that solution fail while Storage Spaces flakes out. Super odd software issues, disk rebuild issues after failure, etc. Basically everything you'd fear going wrong with a software-based solution (especially in Windows...) vs hardware RAID.

2) Hardware RAID. The issue here is drive group & logical drive limits on RAID controllers. For instance, on LSI controllers (which I typically swear by) you're capping out at 32 HDDs per drive group. I'm certainly not gonna stripe across 32 x 14TB drives, and that would still cap me at just over 400TB usable.

3) QNAP ZFS NAS, presenting iSCSI to a Windows server. Won't do it due to iSCSI limits.

4) Hardware RAID, with multiple RAID-6s (or 60s) presented to Server 2016, then stripe across them using disk management to create one large ReFS volume. This one intrigues me but I'm not sure how Windows will react if it can't access the drives directly due the RAID arrays. I'm curious if anyone's tried this.

5) Linux (in general). I think the issue here will be the application that's going to run on the server, which I believe is only supported on Windows.

Wondering if anyone has any other idea.

 

[mdd1963]

quite a few options for those willing to spend $20-$25K ...just on drives to put in the server.

https://www.45drives.com/products/

 

[marko55]

quite a few options for those willing to spend $20-$25K ...just on drives to put in the server.

https://www.45drives.com/products/

These look like typical JBOD setups, utilizing traditional HBAs or RAID cards. That classifies them in bullet 2 in the original post, which suffers from limitations mentioned unless I'm missing something.

 

[failboat]

Gluster can be mounted to many systems. It's most common to mount into a SAMBA share and then mount that on each client, but you can mount Gluster on each client if you really want. Either way it's all the storage you want. I believe the client calculates the shards so it's probably best to run a SAMBA share.

Build out 16+8 vols distributed dispersed. Each server will have 4 disks per vol.
6 servers 2 can go down and the storage is still up. 9 disks from the same vol would have to fail before any data loss.

12 TB disks = 192TB per vol
If your servers are sized to handle 24 disks each then you could have up to 6 vols which is over a PB.

I'd probably find someone who builds these and buy it. Ceph works in a similar manner. I'm fairly certain more people use Ceph for very high storage pools. Figuring out what hw to buy and how much ram etc will take a pro. I wouldn't DIY this.

Large cloud enterprise storage solutions have a similar setup except the nodes are spread out into geographic regions. Which is a lot less risky than everything under 1 roof. You might be able to buy rack space in separate places and hire people to service it.

 

[mdd1963]

Something that large is going to have RAID 50/60 or a cluster of 3 ore more systems...