48Tb NAS - Your thoughts? Or a custom build of up to 72Tb?

Toggle sidebar Toggle sidebar
H

henri br

Distinguished
May 3, 2011
17
0
18,510
Thecus N16000

We've been working on project which may require us to share a few hundreds educational contents/files in audio, video & book formats. We'd like to share these contents on eMule/bitTorrent similar apps, as well through HTTP/FTP. Also, as we've been in need of a network-attached storage solution I thought this one would be a solution for all these needs. The problem is that this NAS seems to be in "beta version": 1, 2, 3, 4.

The NAS requirements is something that can:

- Handle 15 local persons editing and messing these contents
- At the same time, share these contents on eMule/bitTorrent similar apps, as well through HTTP/FTP
- Support Sata III 3Tb HDDs like this Hitachi
- Support 10Gb Ethernet card
- Any 2U, 3U or up to 5U NAS would be fine
- Be someway future proof, and provide at least 48Tb of storage

This is not a mission critical system, and we are not worried on uptime.
It's more to local use, P2P sharing on a 200Mbit WAN...
The critical part of this project may be hosted somewhere "in the cloud".

What are your thoughts about this Thecus N16000 NAS?
Do you know some other that would fit on above requirements?
Any advice on these as I'm a newbie on this subject?


Thanks, and by the way it's nice to become part on Tom's Hardware! :)
 
Solution
F
OK - Want to pay me to build it for you :) LOL

Here this is I would do - base on my experience.

Intel MB with at least TWO PCI Ex 16x V2.0
CPU - your pick. There is not much of CPU usage when run as NAS
Mem - 2~4GB is good enough
16bay 3U rackmount server chassis
500W Redundant PSU (check the 12V line make sure it rates at 28.0A min)
16x port hardware raid SAS6 controller: ARECA, ADAPTEC, HTP, LSI, 3Ware
- I always used ARECA, since I rarely need support from them, but you may choose US base manufacture
16x 3.0TB HDD Ultrastar
10Gb Chelsio Ethernet card
1x USB thumb drives - 4GB

Software
1st choice: FreeNAS
2nd choice: OpenFiler
3rd choise: Chelsio USS
4th choice: Open-E
5th choice: 2008 server

FreeNAS
Create a boot CD
Load...
Sort by date Sort by votes
F

FireWire2

Distinguished
Oct 30, 2006
1,304
1
19,665
Henri

Sans Digital AccuRAID AR424I is iSCSI box it wont support HTTP/FTP. File server and Bittorrent, so as AccuRAID AR316I

BTW what do you mean of:

- Handle 15 local persons editing and messing these contents
Are we talking about editing/messing with SD or HD content?

Let me do a little research I will post later where you go or call...
 
Upvote 0 Downvote
H

henri br

Distinguished
May 3, 2011
17
0
18,510
Thanks for clarifying FireWire2!
I've been visiting several storage solutions providers and some forums...
But, I'm still quite lost on this as I'm not literate on these subjects.



We are group of 5 persons working on several projects that overall, are related to culture and education.
There are some other projects that we help to maintain.
Frequently, we count on other 10 to 15 persons helping us on our daily activities.
These persons should have access to the NAS on the LAN.

Probably, we'll need that to provide remote access as well on a "root level" at some cases; acess to everything.

On this August, we are going to start to receive a mass amount of files in video, audio, and book formats to be reviewed.
Up to 25 persons will be editing these files, and make them ready for free distribution.
There will be some HD contents; so yes, we are talking about "editing/messing with some SD or HD content".

This distribution will be accomplished mainly through P2P file sharing, and some files through HTTP/FTP.
HTTP/FTP is not our main focus, and we can pay a hosting providers for this.
When it comes to the structure for P2P sharing, we'll be distributing the files through a 200Mbit FTTx connection.

We still don't have the routers, switches and other network equipment for this. Probably, we'll buy some Cisco stuff.
So, we'd like to plan it properly and hear our friends in Tom's Hardware community to do our best with regard to:
- Buy the right "things", keeping in mind the future!

The solution will be also used for our storage/back up needs; and maybe for learning about virtualization technologies (Hyper-V from MS, VMware, etc).

I've edited the first post, as I did some mistakes there:
It is an at least 48Tb solution.
It can be a 2U, 3U, or up to 5U rack mount solution.
 
Upvote 0 Downvote
F

FireWire2

Distinguished
Oct 30, 2006
1,304
1
19,665
How about an Scalable NAS/SAN you can expand-able up to 128 HDD, support HDD > 3.0TB and multiple volumes

This will allows you to:
Expand the existing volume with the same size HDD, or
Create a new volume with a future HDD, which maybe greater than 3TB
 
Upvote 0 Downvote
H

henri br

Distinguished
May 3, 2011
17
0
18,510
There would be no problem to build a solution from the scratch, since we can have some help.
The time and patience we already have... :)

That's it fellow:
An scalable/expandable NAS solution.
A SAN would not work for us as it is somehow expensive and would require much more money than a NAS.

I don't see many solutions providing support for SATA III and SAS 6G HDDs. Why?
Also, a solution with SSD support for the O/S is a huge plus, no?
 
Upvote 0 Downvote
F

FireWire2

Distinguished
Oct 30, 2006
1,304
1
19,665
OK - Want to pay me to build it for you :) LOL

Here this is I would do - base on my experience.

Intel MB with at least TWO PCI Ex 16x V2.0
CPU - your pick. There is not much of CPU usage when run as NAS
Mem - 2~4GB is good enough
16bay 3U rackmount server chassis
500W Redundant PSU (check the 12V line make sure it rates at 28.0A min)
16x port hardware raid SAS6 controller: ARECA, ADAPTEC, HTP, LSI, 3Ware
- I always used ARECA, since I rarely need support from them, but you may choose US base manufacture
16x 3.0TB HDD Ultrastar
10Gb Chelsio Ethernet card
1x USB thumb drives - 4GB

Software
1st choice: FreeNAS
2nd choice: OpenFiler
3rd choise: Chelsio USS
4th choice: Open-E
5th choice: 2008 server

FreeNAS
Create a boot CD
Load FreeNAS to USB as embeded OS
Configure the RAID per your desire RAID5/5, hot spare
Boot from just created USB
Log In FreeNAS
Mount the raid volume
Configure SMB/NFS/FTP/Bittorrent services
Fine tuning the SYTEM change MTU, tcp, udp...

I like FreeNAS because it's an open source. Help is readily available in it forum, just like Tom's Hardware forum

I have not try 10Gb Ethernet with FreeNAS, but Open-E. I think it should work, because FreeBSD supports it

Most of Linux/Unix base NAS runs from RAM after it boot, and it normally runs for months so there is no need to use SSD.

If you are using 2008 server then you may consider SSD


Hope this will get you start
 
Upvote 0 Downvote
Solution
H

henri br

Distinguished
May 3, 2011
17
0
18,510
Thanks for being promptly!
What are your rates? :p

I'm not sure if could understand it.
1. Why it wouldn't support? Could you explain the above statemet?



Quite insightful!
1st choice: FreeNAS; yes.
5th choice: 2008 server; it may be useful for learning, so we may try it also.

I thought the AccuRAID main differences was the 10Gb Ethernet and some other better hardware.
2. So a 10Gb Ethernet card would work at no problem for HTTP/FTP/Etc?

I'm going to study a bit more about the hardware you've advised.
3. What kind of hardware would support Sata/Sas both 6 Gbit/s HDDs? And also SDD "embedded"?

Thanks,
Henri
 
Upvote 0 Downvote
F

FireWire2

Distinguished
Oct 30, 2006
1,304
1
19,665
My rate $135.00/hr :), but today is free :)

1_ iSCSI is a network storage, it's centralize the data storage. When mount a iSCSI drive in your system like a local C: or D: drive, but not a network drive... more info go here: http://en.wikipedia.org/wiki/ISCSI

iSCSI does not have networks services such as: HTTP/FTP/SAMBA/NFS....

2_ 10GbE, Bonding GbE, or 10/100Mb NIC they all work for HTTP/FTP/NFS/ATP... It just the available of system services and whether it enable or disable.

3_ Most of the new SAS controllers are SAS/SATA 6Gb, to utilize it the PCI Express slot MUST be V2.0
 
Upvote 0 Downvote
H

henri br

Distinguished
May 3, 2011
17
0
18,510
I've been reading that link on WikiPedia, but I couldn't get into the point and realize it.
Anyway, I'll read it again and again.

1. So, any iSCSI based storage would not fit into requirements, right?



I saw many people talking about this controller card (link) and I can see many other around there; but most does not specify SATA III or 6 Gb/s... That's why the concern. This one I've mentioned seems to support it, but many "built solutions" or even some controllers do not specify it clearly.

2. What about the SSD? Does the SDD HDDs are also plugged on those controller cards? If not, what is needed to also support 2 SSDs for an operating system?
That card on above link seems to support SDD.

Edit: Yes, all SATA/SAS/SSDs HDDs can be connected to a controller card that support it. So, the question two is answered.
Any note about it?



I've been reading this website (link) and a few other to understand how these things works, and what are the possibilities.
That website has a comparision chart:
SAS Expanders vs. SAS (no expander)
SAS Expanders vs. SATA Multilane
SAS Expanders vs. Port Multipliers
BTW: I were not able to fully understand those charts; mainly the diagrams. :??:

Image for reference on the following questions (link)
3. Can those internal SFF-8087 connectors be used for other things than connecting 4 HDDs each? If so, what for?
4. Can I mix SATA/SAS/SSD on each connector with no problem? (curious)
5. What are those connectors in the middle of the back heatsink? What is that used for?

Another image for reference on the following questions (link)
6. The center connector is a SFF-8088. As I understood, it can be used for other 4 HDDs. But, is it also used for connecting to an identical card so we could have another set with other 24 HDDs (making the NAS expand-able)? What are the purposes of that external connector in terms of possibilities?

Entire review (link)

7. What I still don't understand is, if in the future we another set with more 24 HDDs, and then more 24 HDDs; each time we need to extend, do we need to buy all the things again (MB, CPU, Mem, another controller, etc.)? How does it work (I mean, the process of extending to up to 128 HDDs)? How are the "stuff" connected?


Thanks so much for your efforts! :)
 
Upvote 0 Downvote
H

henri br

Distinguished
May 3, 2011
17
0
18,510
Knok, knok... Anyone home? :hello:

Everytime we need a new build, Intel makes us crazy with so many sockets/chipsets.
As we'll use this server to play with VMware, Hyper-V SP1 from Microsoft, or other, I'm wondering what cpu/mb to buy at this time...
And, did I remember to mention TrueCrypt and encrypted volumes?

LGA 1366? What about it's getting retired?
LGA 1155? What are the future's perspectives for this?
LGA 1567? Powerful (low clock speed) processors, quite expensive. Is it worth?
What else?
Bah! What to do, considering the future?

We were thinking about:
X5690 (upgrad-able to 2 processors)
W3690 (single processor option)
or something at E7 family

Mind to share your opinions on this?
 
Upvote 0 Downvote
F

FireWire2

Distinguished
Oct 30, 2006
1,304
1
19,665
Wow, that's lots of quesitons :)

1_ iSCSI wont working with your requirements

As far as SATA III and II you may see a different in the raid itself (speed), but the intend system may have 10Gb E, which is very easy to saturated with the 16/24 drives raid card, regardless SATAII or SATAIII.

Addition to it, the SATA III chipsets with 4 ports are not readily available compare to SAS III...

2_ SSD is very good in OS with HDD access regularly like Windows, but application specific like: Open_filer, Free-NAS, Open-E, USS, Open-Samba... After boot it runs from RAM, therefore SSD is not really benefit

Also, once configured, you can save the configuration to a file... which it take seconds to restore


With FreeBSD/FreeNAS there is ZFS feature which is really cool... the person with the BEST knowledge about it is Submesa. You can use the ARC-1300-4x/i and SAS expander to have indiscriminate HDD pool with up to 128 HDD

Other questions:

miniSAS on SAS controller does have dual purposes: Expanding and HDD connection

a_ you can connect to HDD and the SAS controller will see 4x drives
b_ connect to SAS expander, then SAS controller will see all the HDD connect to the expander and other DAISY chain SAS expander's HDD

If you are using ZFS feature then you get the powerful CPU, other wise save the money for the hw raid controller
 
Upvote 0 Downvote
H

henri br

Distinguished
May 3, 2011
17
0
18,510

It's true... lol
And you've been of great help here!


I'm researching the parts for this rig, and now the question is:
1. Which 10GbE (dual port) should we buy? SFP, RJ45? Could someone help on this, and post some links?

Here it is an Intel's link for reference only.
Maybe a helpfull faq here (for those who are able to fully understand it - Not my case =D)
Another Intel's link
Jumbo frames is "a must have", no?


We are going to pick some Cisco parts to build this network.
We don't have none network equipment for this build yet.

The "best" motherboard (for this) I've found seems to be the SuperMicro X8DAH+ (link to specs).
It is a LGA 1366 - Maybe I'm wrong about this choice, considering my last post; I'm not sure.
 
Upvote 0 Downvote
F

FireWire2

Distinguished
Oct 30, 2006
1,304
1
19,665
That is very good MB.

I would use only ONE CPU instead

Look at the Chelsio 10Gb cards. It's VERY GOOD adapter:
http://www.chelsio.com/assetlibrary/t4-docs/DS-T420-SO-CR.pdf
There is low cost 10Gb switch is CX4, but this have limited in cable length, where SFP you can use a long cable (Optical) if need it.

Look at toshiba 10Gb swithes... they are very decent

With 16.24 ports SAS controller then you are good to go



 
Upvote 0 Downvote
H

henri br

Distinguished
May 3, 2011
17
0
18,510
Yes, we are going to buy only one CPU.
When, and IF needed, maybe we buy a mate for him.
Anyway, that motherboard would give this possibility.

About the network structure...
There is no need for long cables, I mean more than 100 meters long. 20 to 60 meters would be enough.
We still need to study it better, we are going to do it after we decide which server we are going to "assemble".
Note: For those who are also learning about 10GbE, there are some explanation here (link)

Thanks for the Chelsio T420-SO-CR link.
I'm going to study it, and learn more.

And for those Intel's; which would you pick considering you already know what's involved on this project? link
This question is because we're in Brazil, and I'm sure we can get a fast/local support from Intel.


BTW: It seems we are going to have not only a NAS, but a multi-purpose server, which NAS is the main one.
And a note on NFS vs CIFS, URL 2 :)
 
Upvote 0 Downvote
F

FireWire2

Distinguished
Oct 30, 2006
1,304
1
19,665
I would go for the CAT6 Intel® Ethernet Server Adapter X520-T2. But you need to check out the 10Gb E switches with CAT6 first

as far the NAS OS goes:

FreeNAS, OpenFiler, Open-E, USS - does offer both iSCSI and NAS functions
So it can be use as Multi-purposes Server.

 
Upvote 0 Downvote
H

henri br

Distinguished
May 3, 2011
17
0
18,510
Until now I thought of going with the following hardware:

01x ... Motherboard: SuperMicro X8DAH+ (link)
01x ... Processor: Intel Xeon X5690, Westmere-EP (link)
02x ... Memories: CT3KIT102472BB1339 3x8 (link)
01x ... Raid card: Areca ARC-1880ix-24-4G (link)
01x ... 10GbE: Intel X520-T2 (link)
01x ... Video card: Nvidia Quadro 4000 (link) - Yes, we may use the server directly for HD video edition
20x ... HDDs: Hitachi HUA723030ALA640 3Tb 6Gb (link)
03x ... SSDs: Intel 510 120Gb (link) - In raid 5 for O.S./virtualization, maybe?
01x ... Optical drive: Sony Optiarc BD-5740H Slim (link)

1. Am I missing/forgetting something? Should something be included?
2. Will above hardware work for what we are proposing?
3. What would you change in it?

I know it is not included the power supply, as we'll need to sum the power requirements; but I can't remember of other parts right now.
LCD monitor and chassis we'll buy from a local dealer.

Anyway, is there something we should pay attention when it comes to the storage chassis/enclosure?
4. Anything special with it? We have no idea on what kind of thing it requires.




I'm going to start doing it today. I'm going to check some switches.

Thanks A LOT for your GREAT assistance FireWire2!




Parts, quick specification:
--------------------------------

Motherboard: SuperMicro X8DAH+ - Full description link
Dual 1366-pin LGA • dual 5520 Chipsets • 288GB DDR3-1333 • Intel 82576 Dual-GigE • 6xSATA2 via ICH10R • 1 EIDE, ALC888 7.1 HD Audio • 2x IEEE 1394a • 2 PCI-E x16, 4 x8, 1 x4 (in x8 slot)

Processor: Intel Xeon X5690, Westmere-EP - Full description link
6 cores 12 threads • 3.46 GHz - 3.73 GHz max turbo • 12 MB cache • 2 QPI links • 64-bit • SSE4.2 • max of 288 GB DDR3-1333 3 channels memory • 130 W
*If more than 6 memory modules are in use, the frequency decrease to 1066. I'm not so sure about it.

Memories: Crucial CT3KIT102472BB1339 - Full description
2x 24GB kit (8GBx3) • DDR3 PC3-10600 1333 • CL=9 • Dual Ranked • Registered • ECC • 1.5V • 1024Meg x 72

Raid card: Areca ARC-1880ix-24-4G - Full description
1 x SFF-8088 mini-SAS (external) • 6 x SFF-8087 mini-SAS • PCIe 2.0 x8 • 4GB ECC DDR2-800 • RAID 0, 1, 1E, 3, 5, 6, 10, 30, 50, 60 or JBOD • 6Gb/s and 3Gb/s SAS/SATA/SSD drives

10GbE: Intel X520-T2 - Full description & Comparison link
10Gbps Cat 6A up to 100m Copper • PCIe 2.0 • Intel VT-c (VMDq & VMDc) • NO fiber channel over ethernet

Video card: Nvidia Quadro 4000 - Full description
256 CUDA Cores • 2 GB GDDR5 256-bit • DX 11 / OpenGL 4.0/3.2 • 1 Dual Link DVI-I • 2 DisplayPort • 3D Vision/3D Vision Pro • 142W

HDDs: Hitachi HUA723030ALA640 - Full description
SATA 6Gb/s • 3TB • 64 Mb buffer • 7200 RPM • 512 Bytes/sector • Seek Time (read, ms, typical): 8.2 • Dual Stage Actuator (DSA) & Enhanced Rotational Vibration Safeguard (RVS) • 11.3 W

SSDs: Intel 510 120Gb - Full description
120GB • Read: up to 400 MB/s • Writes: up to 240 MB/s • Random 4KB Reads: up to 20K IOPS • Random 4KB Writes: up to 8K IOPS

Optical drive: Sony Optiarc BD-5740H Slim Full description
BD 6X • DVD 8X • CD 24X • SATA • 5.8 Mb buffer
 
Upvote 0 Downvote
H

henri br

Distinguished
May 3, 2011
17
0
18,510
Hello there,

I've read about your build and started learning exactly with that =D
I've read it also on another forum. That's nice!

I've mentioned that video card because we were going to build a workstation to deal with video edition / rendering.
As our primary role won't be playing with videos, and it'll be someway sporadically we are thinking about not to build it.
A workstation would be somehow costly when compared to just throw a video card on that server.
Then, we may use the server for such "intensive tasks".
Again, I may be wrong about it; I mean, about going this way.
Maybe, a "gaming class video card" would fit perfectly, and do a lot more in general terms than the overpriced Quadro cards.
GTX 580's, 590's, HD 6990... Who knows!?
A GTX 590 (spec. link) is priced for $30 less than that Quadro on NewEgg (link), there's also the AMD HD 6990 (spec. link).
Anyway, I know that Quadro's aims on a different market, has different software/drivers optimized, etc.
Just to remember, budget for a video card shouldn't be more than $800 - "Is it a server!?" :-D
I'm going to study these points further.
1. What do you think?

Some links about it: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, and 11; for those who are interested.



For the SSDs, fellow, I'll tell you :D
The guys here really loves to try every piece of thing, and make changes every now and then.
I'm quite sure that, at some point, we're going to test a lot of things in this server...
So, as I know it, I want to be prepared. :pt1cable:

2. Probably it's a dumb question, but, I actually would like it to be a RAID 10, and not 5 as I've told.
That choice is due to 6 x SFF-8087 on the RAID card: 20 for HDDs, 3 for SDDs and 1 for the Blu-ray burner.
What about the motherboard's ports? That mobo has 6xSATA2...
Will it be deactivated when installed the RAID card? If so, is it possible to keep those 6xSATA "active"? Any drawback on it?
I'm asking because we could "plug" the blu-ray burner on the first port of the mobo, and use that freed port for the 4th SDD: RAID10


Thanks for the link and "SAS 6.0Gb/SATAIII support" notification!
3. What do you mean with "three modules"?



Just a side note, in terms of power consumption, with this current build we're working, you'd be able to build other 4 rigs like yours ;-)

Cheers
 
Upvote 0 Downvote
H

henri br

Distinguished
May 3, 2011
17
0
18,510
Some storage chassis manufacturers I've found...

http://www.supermicro.com/
http://www.chenbro.com/
http://www.aicipc.com/
http://www.dynapowerusa.com/
http://www.norcotek.com/
http://www.polywell.com/
http://www.istarusa.com/
http://www.urack.com.tw/
http://www.arktechinc.com/
http://www.procase.nl/
http://www.pcicase.co.uk/
http://www.habeyusa.com/
http://www.primesource.in/
 
Upvote 0 Downvote
F

FireWire2

Distinguished
Oct 30, 2006
1,304
1
19,665
Three modules redundant gives less stress to the remain module when power fail happen.

Two modules redundant PSU
When run each provide 50% power, when one goes bad the other have to supply 100%, therefore 100% increase power

Where three modules:
One module failed, only 33.3% power is needed to provide from the remain two, therefore each power module only need to increase additional 50% power.

So three modules redundant power supply (RPS) will be more reliable then the the RPS with two modules

 
Upvote 0 Downvote
H

henri br

Distinguished
May 3, 2011
17
0
18,510
@FireWire2,
What kind of HDD are you using in your 40TB Media Server?
Answer: 20x WD Green 2.0TB WD20EARS (link) ;)
I'm OVERloaded trying to learn so many things in a short period, but it is good.


What is a really effective anti-vibrating solution to add in this new(bie) build for the HDDs?
Let's suppose we'd buy this chassis; not sure, however.
 
Upvote 0 Downvote
You must log in or register to reply here.

TRENDING THREADS

Latest posts

Moderators online

Share this page

COMPANY
RESOURCES
FOLLOW
Top Bottom