16-bay RAID and C2750 motherboards

[phunqe]

Hello,

Basically I want to create a 16-bay NAS with the Intel C2750 cpu. To minimize power consumption I am planning to opt for the new Seagate Terascale 4TB drives.
The two suitable motherboards seems to be SuperMicro MBD-A1SAI-2750F and AsRock C2750D4I.

I have contemplated a LSI 9201-16i HBA and using FreeNAS with ZFS, but I feel it's too limiting for my expectations to run different software on the server as well.
An alternative could be just a custom box (e.g. Ubuntu), but is a 16x4TB disk raid a good idea with ZFS (assuming Z2 level or whichever allowed for two disk failures)? I read on the FreeNAS forums that 12 disks seems to be some "reasonable" limit. I know I would need a lot of memory, but 32Gb is the most those cards can take, since there aren't any suitable 16Gb modules yet.

I am perfectly fine with Windows and an Adaptec 71605 card, but then my issue is if that card actually work on those MBs. It would be a major hassle if I had to find out the hard way it's not working. If I am honest this is my preferred choice at the moment, since I run a Windows RAID-6 box already so I know exactly what to expect and how things work. The main problem is, as mentioned, the question of compatibility with the RAID card.

If a custom BSD or Linux box with ZFS and that amount of disks works I would be willing to give it a shot with an HBA card, I saw somewhere that the LSI 9201 should work with at least the SuperMicro board.
I don't have any Windows only applications I need to run and remote X or VNC works for me.

Thoughts/tips/etc are most welcome.

Thank you.

 

[TyrOd]

Hello,

Basically I want to create a 16-bay NAS with the Intel C2750 cpu. To minimize power consumption I am planning to opt for the new Seagate Terascale 4TB drives.
The two suitable motherboards seems to be SuperMicro MBD-A1SAI-2750F and AsRock C2750D4I.

I have contemplated a LSI 9201-16i HBA and using FreeNAS with ZFS, but I feel it's too limiting for my expectations to run different software on the server as well.
An alternative could be just a custom box (e.g. Ubuntu), but is a 16x4TB disk raid a good idea with ZFS (assuming Z2 level or whichever allowed for two disk failures)? I read on the FreeNAS forums that 12 disks seems to be some "reasonable" limit. I know I would need a lot of memory, but 32Gb is the most those cards can take, since there aren't any suitable 16Gb modules yet.

I am perfectly fine with Windows and an Adaptec 71605 card, but then my issue is if that card actually work on those MBs. It would be a major hassle if I had to find out the hard way it's not working. If I am honest this is my preferred choice at the moment, since I run a Windows RAID-6 box already so I know exactly what to expect and how things work. The main problem is, as mentioned, the question of compatibility with the RAID card.

If a custom BSD or Linux box with ZFS and that amount of disks works I would be willing to give it a shot with an HBA card, I saw somewhere that the LSI 9201 should work with at least the SuperMicro board.
I don't have any Windows only applications I need to run and remote X or VNC works for me.

Thoughts/tips/etc are most welcome.

Thank you.

You want to run a 48 TB system with low power Seagate 4TB drives?
That's just a really bad idea unless you have a really robust backup and are okay with significant downtime to restore from backup.

I wouldn't run anything less than a RAID 10 and preferably something like RAID 50/60 with hot spares.

Also...how are you planning on backing up 48TB?

 

[phunqe]

I won't backup 48TB. A lot of this will be video material, which I isn't reasonable to backup. The data I need to backup, which will be a few TBs only, will be backed up regularly to an external USB drive and also online.

As for the low power disks, from the benchmarks I've seen it's not that bad and I am not looking to build high performance array in any case. I do understand the concerns brought forth however and I'll keep them in mind.

Thank you.

 

[TyrOd]

I won't backup 48TB. A lot of this will be video material, which I isn't reasonable to backup. The data I need to backup, which will be a few TBs only, will be backed up regularly to an external USB drive and also online.

As for the low power disks, from the benchmarks I've seen it's not that bad and I am not looking to build high performance array in any case. I do understand the concerns brought forth however and I'll keep them in mind.

Thank you.

It's not really the performance that was a concern. High density, low power drives like these in a NAS environment have a ASTRONOMICAL failure rate. 16 of those drives,, even in a RAID 6 is going to be really unstable and create a lot of downtime.

Whatever data you're not backing up in a system like that you should expect to lose. It's just a matter of time.
Even with RAID 50/60 it would be a huge pain in the ass to be constantly replacing drives and rebuilding the array.

 

[phunqe]

When you say they are unsuitable for a NAS environment, do you mean that they are rather only suitable for cold storage?

What do you in that case think of the new HGST NAS drives? They have vibrational sensors and are have a higher MTBF rating (which I know should be taken with a grain of salt obviously, but still).

Or is the WD RE or the Seagate ES drives the only option for an array this size?

 

[TyrOd]

When you say they are unsuitable for a NAS environment, do you mean that they are rather only suitable for cold storage?

What do you in that case think of the new HGST NAS drives? They have vibrational sensors and are have a higher MTBF rating (which I know should be taken with a grain of salt obviously, but still).

Or is the WD RE or the Seagate ES drives the only option for an array this size?

They hold up poorly in 24/7 uptime environments.

I would strongly suggest NAS/Enterrpise drives because of their feature set.

HGST would be my personal preference.

 

[phunqe]

Thank you for your suggestions and tips, much appreciated. I'll keep everything in mind while I get more grey hair trying to decide what to do 😛

Cheers.

 

[phunqe]

I found an interesting link http://www.servethehome.com/raid-calculator/raid-reliability-calculator-simple-mttdl-model/

By entering data there you get quite a good picture of how vulnerable a raid with many disks is. By studying the differences I am now moving towards having two 8 disk raid 6 arrays instead.