Help with DAS/NAS diy

[BLACKBERREST3]

Hello, I am looking to build a DAS/NAS. I have been researching on what the best solutions would be for performance and it has left me with many questions. I would like to use existing hardware if possible which consists of: i7 6700k, 64gb DDR4 (non ecc) corsair ram, Z170 Deluxe (20 lanes all together). I will use SyncBackPro for scheduled backups. I want insane read/write speeds on the main work portion (around 8 terabytes) and 2 high-speed redundancies (I’ll add more as I need it). I am looking towards using software raid and using either FreeNAS or Windows Server 2016. I also need my data to be byte perfect with no degradation over time to preserve precious data. I plan to use this as a personal DAS/NAS and not something that would require it to run all the time.

My questions are:
1. ZFS or ReFS, suggestions?
2. Can I use ram as a non-volatile super cache or lazy read-writer if it is always powered and I keep redundancies to prevent data loss?
3. What is the best set up for performance that also lets me add more storage easily if I need it; raid0 SSDs + raid10 HHDs or tier-ing or something else?
4. What SSD/HDD combros do you guys recommend, I am leaning towards seagate for HDDs?
5. If a raid array fails, does that mean that I must replace the drive that failed or all drives because it damaged it somehow (only talking about hardware not data)?
6. What is the best way to connect to this DAS/NAS; direct PCIE pc to pc or 40/100gbe or something else?
7. How would I set up a 40/100GBE connection and what would I need?
8. Is there any other thing that I may need to know or want relating to this?

 

[the nerd 389]

If you used clients as the processing horsepower, you'd be limited by network latency more than IO latency. You'll need to choose between IOPS and sequential performance south of 5 digit builds. Regardless, hardware RAID is still faster than software RAID. Hardware RAID uses ASICs for the processing grunt. CPUs can't match them, but can come reasonably close in some workloads.

There are no decent, inexpensive options for 8 TB of SSD space. Most commercial servers would try to employ some sort of tiering system to improve performance without breaking the bank. You'd have to rethink the workload, as full-on sequential loads will not do well on a tiered system. If you can break the work up into smaller pieces, it can make quite a difference, though.

I'd avoid JBOD unless you have a very good and well-documented reason to consider it. Microsoft Exchange is a good example. If you have no other choice but to go for solid state, you'll have to either get enterprise-grade write-centric SSDs and put them in RAID (and suffer a performance hit), or you may have to resort to one of the software-defined-storage approaches.

I've never dealt with SDS personally. It's a completely different beast than RAID, and I've never hit the limits of RAID. It's significantly more capable, but also much more involved to deploy.

Lastly, I'd try not to worry too much about the backup array's speed. Instead, focus on reliability for that block. Speed is much more expensive than reliability. Often, they prove to be mutually exclusive outside of a datacenter.

 

[USAFRet]

Here's an example of what a small dedicated NAS can do.
Qnap TS-453A
Celeron N3160 @1,6GHz
16GB RAM
4 x 3TB WD drives in RAID 5.
Standard 1gigabit LAN, through 2 switches and the router.

Currently it is:
Playing a movie (1984) out to the TV
Playing the music lib out to a different PC
Receiving 165GB of data (4 x .VDI files) from my main Win 10 PC

CPU utilization hovering around 35%
Nary a burp or pause in the movie or the music.

 

[BLACKBERREST3]

I have done some digging on my next build and I need some help on this. Z10PE-D16 WS (mobo), 2x Intel® Xeon® Processor E5-2637 v4, Crucial 64GB Kit (4 x 16GB) DDR4-2400 ECC UDIMM, AX1500i PSU, and the Phantom 820. nvme is a new technology after all. I think I will settle for sata 6gb/s speeds from an 8tb ssd instead and upgrade when technology gets better. The only problem is that I will need to use another LSI HBA in the main build too to prevent a bottleneck (DMI 2.0) because I use a lot of usb devices and peripherals too.

1. Will the Z10PE-D16 WS (SSI EEB) fit in the Phantom 820 (ATX, E-ATX, XL-ATX)? I heard the mounting points were slightly different.
2. Will this memory work and do I need to worry about ranks if I plan to install more than 128GB?
3. Anything else I should look out for in a dual socket setup?

Didn't see past posts before I posted, my b;
As long as I get the best bang for the buck, I am willing to go upwards from 10-15k if necessary. I chose Seagate for storage because they are the most reliable in my experience. Samsung on the other hand is where I go for performance. I said before that I'd rather build it because it is scale-able, customize-able, and upgrade-able, ironically, I was originally looking at the thunderbolt 3 qnap before I decided to build one. What do you mean by network latency? I have symmetrical gigabit internet for remote storage if that's what you mean. You are definitely right about SDS. When I was reading about it, half the stuff went over my head. I'm probably going to try to delve into it again because it is a part of windows 2016, and if I'm going to use ReFS then I might as well get the most out it. Just a standard licence is 882$ though.

 

[the nerd 389]

I'd go with something like the X10DAi from Supermicro. Regarding the case, consider getting a rackmount. It makes life much easier, but you'll have to deal with loud fans.

You can build the rack to include ample backup power, networking gear, and it allows you to ensure you have room for any expansions down the line. I'd suggest getting a half-height unless you want to put the monitor and keyboard inside the rack.

Note that you will have to get fairly creative with cable management to get the most out of a rack. Specifically, the fans in most server chassis are hot-swap. In order to swap them, you have to slide the server forward like a drawer out of the rack to access them. Ensuring the cables are sufficient length for that and don't get tangled takes planning.

Also bear in mind that most server hardware is designed with the idea that it will be in an isolated room. Acoustics are generally ignored. Expect extremely loud fans. I'm not talking about "loud air cooler" levels, either. I'm talking "Do you have a small jet engine in the other room?" levels of noise.

Supermicro makes some good server chassis, especially at the entry level. If you get one with a PSU, you can find options up to redundant Titanium 2000W units. You probably only need around 800 watts, though. Some of the larger chassis support 4-way redundant PSUs, but that's rather excessive in this build. I'd use 2-way at the most. The specific chassis I'd point you towards is the CSE-846TQ-R1200B . You can find more information on it here:
https://www.supermicro.com/products/chassis/4U/846/SC846A-R1200B

That RAM kit should work, but you'll be limited to 2 channels per socket with it. If you want 64 GB, I'd recommend an 8x8 GB arrangement instead. That'll give you enough to use all 8 channels.

I've had decent experiences with Seagate. I've got more experience with HGST and WD, though. That choice largely comes down to preference, though. I can't comment on which drives from Seagate are appropriate at the moment. If you were considering WD, I'd point you to the Gold and RE lines. The Red Pro is the lowest end drive that supports >8 drive setups. Other drives may die prematurely due to vibration.

Our earlier warnings about RAID and SSDs were meant to be cautionary, and that is the rule of thumb to follow. However, some SSDs are designed for it, but those are a special case. They are not consumer drives, and they are not your common TLC flash. NVMe drives haven't really gotten to that point, yet, unfortunately.

Regarding Samsung SSDs, the only model I'd recommend for this workload is the PM863a. The PM863a can handle RAID, as can a small number of SSDs in a similar class from other manufacturers. If you combine these with hardware RAID, you might be able to get a cheaper pair of CPUs.

Very thoroughly research how well threaded the software is that you intend to use. If it's scaleable enough, consider the E5-2630 V4 instead. It does use slower RAM, but that shouldn't be an issue with a total of 8 channels available. It also offers a but more compute power than the 2637 in the right workloads.

 

[BLACKBERREST3]

I don't know if your catching my drift here. The next build I am doing for the main pc is different from the server. The server will be running the i7 6700k on a z170 deluxe. The main pc will be a high-end workstation with an ample amount of bandwidth for any add on card I want to add. I'm not using dual xeons for the server, I am using them for the workstation. The workstation will have the nvme drives while the server is just for storage. It should be simpler this way.

For a full comparison;
(v4) E5-2630/2637
10/4 cores - difference in parallel computing.
3.1/3.7 Ghz - difference in per thread computing (more suited for a workstation).
25/15 smart cache - I don't know how to tell how much cache I would need. At what point does the size of cache not return in performance?
8/9.6 GT/s (16/19.2 GB/s) - I don't know if this is what caps the bandwidth to the ram or if I read the wrong way to calculate qpi bandwidth.
68.3/76.8 GB/s Memory bandwidth (DDR4 2400) More bandwidth, more upgrade-ability (depends).

The E5 2637 v4 is more suited for single/quad threaded performance which is what I would be using it for, plus it slightly outperforms in qpi and memory bandwidth. Unless the cache plays a significant role, I don't need 10-20 cores most of the time. I may use it for video editing or virtualization, but I mainly keep my work unzipped and preserved.

I was thinking of getting an 8tb ssd to crack down on latency because any raid adds some overhead, but I would also consider a pcie 3.0 x16 nvme host bus adapter if I could ever find one. I did find the MZPLL6T4HMLS which is rated for 58,400tb over 5years. There may be a new interface along the road so I was thinking that an all in one drive would be better than a HBA card for newer drives anyway. 6.4tb isn't what I was looking for, but I could try to manage it. If I ever need more space in the server, I will buy a completely different node with a rackmount, xeons, and hardware specifically meant for servers, but until then I will use what I got (6700k, z170deluxe). Power redundancy is not that necessary. The pc-d8000 can have 2 PSUs, but I wouldn't run the server 24/7. As far as the memory goes, the max amount that the Z10PE-D16 WS can handle is quad channel 2400 DDR4. My questions are;

[strike]Do I need to worry about registered vs unbuffered memory if I use over 128GB of memory? unbuffered has less latency, but registered has more stability.[/strike] Just found out that only registered memory is supported on the Z10PE-D16 WS.
Will the Z10PE-D16 WS (SSI EEB) fit in the Phantom 820 (ATX, E-ATX, XL-ATX)? I heard the mounting points were slightly different.
What would 8 channel ram do for me if the max amount of ram is registered ecc quad channel 2400 DDR4?

edit: So I went on a rant about the 2637, but then I was looking into virtualization and that got me interested. So now I'm debating between the 26 37/43/67/87. I also found the ram I'd want to use (KVR24R17D4K4/128).

 

[Rogue Leader]

So I read through this whole thread and I am still wondering what in the world you are doing that you need such massive storage and horsepower. Anyone? Did I miss something?

 

[the nerd 389]

So I read through this whole thread and I am still wondering what in the world you are doing that you need such massive storage and horsepower. Anyone? Did I miss something?

You read correctly. For whatever reason, that's not up for discussion.

I don't know if your catching my drift here. The next build I am doing for the main pc is different from the server. The server will be running the i7 6700k on a z170 deluxe. The main pc will be a high-end workstation with an ample amount of bandwidth for any add on card I want to add. I'm not using dual xeons for the server, I am using them for the workstation. The workstation will have the nvme drives while the server is just for storage. It should be simpler this way.

For a full comparison;
(v4) E5-2630/2637
10/4 cores - difference in parallel computing.
3.1/3.7 Ghz - difference in per thread computing (more suited for a workstation).
25/15 smart cache - I don't know how to tell how much cache I would need. At what point does the size of cache not return in performance?
8/9.6 GT/s (16/19.2 GB/s) - I don't know if this is what caps the bandwidth to the ram or if I read the wrong way to calculate qpi bandwidth.
68.3/76.8 GB/s Memory bandwidth (DDR4 2400) More bandwidth, more upgrade-ability (depends).

The E5 2637 v4 is more suited for single/quad threaded performance which is what I would be using it for, plus it slightly outperforms in qpi and memory bandwidth. Unless the cache plays a significant role, I don't need 10-20 cores most of the time. I may use it for video editing or virtualization, but I mainly keep my work unzipped and preserved.

I was thinking of getting an 8tb ssd to crack down on latency because any raid adds some overhead, but I would also consider a pcie 3.0 x16 nvme host bus adapter if I could ever find one. I did find the MZPLL6T4HMLS which is rated for 58,400tb over 5years. There may be a new interface along the road so I was thinking that an all in one drive would be better than a HBA card for newer drives anyway. 6.4tb isn't what I was looking for, but I could try to manage it. If I ever need more space in the server, I will buy a completely different node with a rackmount, xeons, and hardware specifically meant for servers, but until then I will use what I got (6700k, z170deluxe). Power redundancy is not that necessary. The pc-d8000 can have 2 PSUs, but I wouldn't run the server 24/7. As far as the memory goes, the max amount that the Z10PE-D16 WS can handle is quad channel 2400 DDR4. My questions are;

[strike]Do I need to worry about registered vs unbuffered memory if I use over 128GB of memory? unbuffered has less latency, but registered has more stability.[/strike] Just found out that only registered memory is supported on the Z10PE-D16 WS.
Will the Z10PE-D16 WS (SSI EEB) fit in the Phantom 820 (ATX, E-ATX, XL-ATX)? I heard the mounting points were slightly different.
What would 8 channel ram do for me if the max amount of ram is registered ecc quad channel 2400 DDR4?

edit: So I went on a rant about the 2637, but then I was looking into virtualization and that got me interested. So now I'm debating between the 26 37/43/67/87. I also found the ram I'd want to use (KVR24R17D4K4/128).

That SSD works.

I'm not sure that's a good motherboard for your application. Asus isn't particularly established in the multi-socket space. I'd recommend Supermicro or Tyan. They've been reliable options in the server/workstation space since Asus has been in the consumer space. Also, the dual socket boards are 4 channels/socket for 8 channels total.

Regarding the CPU, all I was suggesting is that you look into how scalable the software is. If it responds well to more cores, you can save roughly $1000 on the CPUs for similar overall performance.

Regarding the case, using a consumer case without explicit support for server boards can be quite risky. You could always get a 4U chassis and use it as a pedestal. Supermicro also offers (mostly) standard cases for this sort of build as well. Take a look at the SC732. If you want a flashier option, there's always the SC732G.

If you'd like, I can try to see if the bolt patterns for that case line up with the SSI EEB standard, but it'd be a purely visual comparison (measuring distances on images).

 

[BLACKBERREST3]

Sorry I've been mia, I was researching the best ways to utilize the drives (thinking about 2x raid 5 or 6) and I think I am going to have to play around with windows storages pools/spaces. I am especially confused about the differences between hyper-v and v-sphere and which would be a good hypervisor if I were to set up VDIs from my workstation. This is a question for another time though because I think I have the final build specs for my system.

 

[BLACKBERREST3]

How does this look;

Workstation
Z10PE-D16 WS
Phantom 820 – have 1
2x XEON E5-2687W
2x KINGSTON 128GB 2400MHz DDR4 ECC Reg CL17 (KVR24R17D4K4/128)
AX1500i PSU - have 1
2x Samsung 4TB 850 Evo 2.5" SATA III SSD Raid 0 - I'll upgrade to PCIE or something else when it becomes cost effective.
2x ST4000DX001 - have them.
2x H110i GTX (CW-9060020-WW) - Already have 1, but to get another I will have to buy it refurbished.
LSI 9305-24i HBA
960 PRO with WINGS PX1 HBA
Nvidia GTX 970 – have
Xonar essence stx – have


Server
Z170-Deluxe - have
i7 6700k - have
2x Corsair DP 32GB (2x16GB) DDR4 3200 C16 (CMD32GX4M2C3200C16) – have 1, but I might not buy the other if I don’t need it.
AX1500i PSU – Need to buy another 1.
20x Seagate Archive HDD 8TB SATA III
LSI 9305-24i HBA
NZXT Kraken x61 280mm - have
PC-D8000


From what everyone says about ssi ceb boards, the holes don't line up that well and even if it can hold there, it would put too much pressure on the board the more stuff you add. I could always try and if it seems like there is too much load on the mobo, then I could get a 4u chassis like you said and build a pc desk to house it.

To answer rogue leader's question, I am going to be using it for an all in one workstation to help me with virtualization projects, data management, video editing, and yes gaming too. I need it to have many pcie slots for the gpu, sound card, pcie ssd, possibly a nic, and add anything else to it later down the road. I also plan to recycle this build into a second node or turn it into the main node for more storage in the future. It will eventually be re-purposed for what it was intended for.

edit: I completely forgot about the interface, I still need a NIC. When I look for 40/100gbe, It seems like companies advertise them as (example) having 2x100gbe running on a pcie3.0x8. Is there something I'm missing or do NICs transfer data faster than the bandwidth provided?

 

[the nerd 389]

How does this look;

Workstation
Z10PE-D16 WS
Phantom 820 – have 1
2x XEON E5-2687W
2x KINGSTON 128GB 2400MHz DDR4 ECC Reg CL17 (KVR24R17D4K4/128)
AX1500i PSU - have 1
2x Samsung 4TB 850 Evo 2.5" SATA III SSD Raid 0 - I'll upgrade to PCIE or something else when it becomes cost effective.
2x ST4000DX001 - have them.
2x H110i GTX (CW-9060020-WW) - Already have 1, but to get another I will have to buy it refurbished.
LSI 9305-24i HBA
960 PRO with WINGS PX1 HBA
Nvidia GTX 970 – have
Xonar essence stx – have


Server
Z170-Deluxe - have
i7 6700k - have
2x Corsair DP 32GB (2x16GB) DDR4 3200 C16 (CMD32GX4M2C3200C16) – have 1, but I might not buy the other if I don’t need it.
AX1500i PSU – Need to buy another 1.
20x Seagate Archive HDD 8TB SATA III
LSI 9305-24i HBA
NZXT Kraken x61 280mm - have
PC-D8000


From what everyone says about ssi ceb boards, the holes don't line up that well and even if it can hold there, it would put too much pressure on the board the more stuff you add. I could always try and if it seems like there is too much load on the mobo, then I could get a 4u chassis like you said and build a pc desk to house it.

To answer rogue leader's question, I am going to be using it for an all in one workstation to help me with virtualization projects, data management, video editing, and yes gaming too. I need it to have many pcie slots for the gpu, sound card, pcie ssd, possibly a nic, and add anything else to it later down the road. I also plan to recycle this build into a second node or turn it into the main node for more storage in the future. It will eventually be re-purposed for what it was intended for.

There are more than a few issues. In fact, I'm having trouble finding a single part that's a good choice for your application. The only possible exception is a single (not double) kit of 32GB RAM for the server. Even that's a stretch.

You're going to butcher those 850 EVOs. In return, they will seize, thrash, and scream in retaliation. You'll get better performance from old-school 10k Raptors in a RAID array. Even standard 7200 drives would be a step up. Those are on the list of SSDs that RAID will eat alive, just like every other SSD on Tom's Hardware except the P4800X and 750. You should be looking on Tom's IT Pro.

The 960 Pro is still a consumer drive. Do not subject it to sustained, write-heavy workloads. As a boot drive, it will work, but it's not something I'd put in a server or workstation of this caliber.

The ST4000DX001 isn't good for sustained sequential workloads (at all, ever), either. Most SSHDs will seize and thrash in sequential workloads. If that's what you had before, it explains a lot. WD Reds would be much better. Some laptop drives would be better. A Celeron could fully load these in sequential compression workloads. I wouldn't be surprised to see an Atom fully load them. A 4-core ARM would be a good fit for those in a sequential compression workload.

Those Archive drives use SMR technology. They're slower than mud. I wouldn't even put them in a cold storage server, as the backup times would be too long. Some people use them for exactly that, though. I'm just not that patient.

The H110i GTX is not suitable for use in this type of build. Stick to air coolers. The x61 shouldn't be used in a server either. Even if the H110i GTX were reliable enough for a workstation like this, it doesn't support Narrow ILM mounting. You can't use it with dual socket boards.

For the money, I wouldn't consider the HBA for the workstation. There are solid RAID cards for that type of build, and you could get entry-level hardware RAID and an expander for that price in the server.

Registered RAM is a waste of money for this build. Unless you have a specific workload in mind that will exceed 128 GB per socket, stick with unbuffered. If the motherboard won't support that, get a different board. Seriously, it's just a bad way to spend your money.

The server's case isn't a good option for that many drives. The odds of drives failing is quite high when you run that many. Hot-swap bays will make the situation bearable. The Lian-Li case will make maintenance quite a chore.

The AX1500i is a good PSU, to be sure. I'd never put that in a server myself, though. It's over-priced, and it doesn't support PMBus monitoring. Get a server grade PSU and use it with the workstation.

The GTX 970 doesn't belong in that workstation. The rest of the build won't play games well at all. It's not intended to. Get a Quadro or Firepro if you want a GPU suitable for the workstation, and actually need it. Those will handle virtualization much better than the 970.

The Xonar is also quite out of place. I don't think it will cause you any particular issues, but it's certainly not something I've ever seen or considered for a build like this.


Now for the advice one what should happen:

Regarding the RAM and motherboard, I've already pointed you towards Supermicro and Tyan. I will do so again. Find yourself a board that doesn't require registered RAM, and get that instead. Then, get two 128 GB kits of unbuffered ECC RAM if you actually need a total of 256 GB. That's 16 sticks of 16 GB each.

Take the money saved on RAM, and put it into the parts that you're actually interested in: the SSDs. Check out the Tom's IT Pro reviews for guidance. I would point you to the SN150 in the AIC form factor or the DC P4500/4600 in the U.2 form factor. If those are too expensive, check out some mid-endurance options. Do not even think about consumer SSDs. They will not give you what you're looking for. Consider the WD Reds before thinking about a consumer TLC drive.

Regarding the SSHDs, throw them out. They're useless for the workload you're describing. If it's as sequential as I think it is, a magnetic tape drive will outperform the ST4000DX001.

For the workstation, you have no reason to get that HBA. None at all. If you convert it to a node down the line, just get an expander then. For this build, you just need an 8-port mid-range RAID card.

Since you don't have the ability to run performance verification trials on the PSU, stick to established server brands like Supermicro.

A pair of E5-2637s would be able to move data around significantly faster than any other component in the workstation build, even in heavy compression workloads. You should have a well-grounded, well-documented reason for getting a faster model than that.

If you want to have a gaming build, get one and put it in the Phantom. Something like your i7-6700k is a good fit. It will be faster in games than the workstation.

For the workstation, don't bother with the 820. Get a decent pedestal case that's designed for dual socket boards. Many 4U cases are designed to be used in either a standard tower configuration or on a rack. For the tower configuration, you put the included feet on it. For a rack, you put the included rails on it.

For the amount of money you're putting into this, you should really allocate it to the parts that matter. This does not include compute server CPUs, registered RAM, M.2 HBAs, consumer SSDs, SMR HDDs, consumer cases, or consumer PSUs.

Lastly, none of the workloads you've described could leverage the CPU resources you've listed with the possible exception of "data management", and that would need to be incredibly unusual to require the compute power you've listed. Those CPUs don't work well with video editing, and will perform marginally better than an i7 6900k. Dual socket builds don't edit videos well. They can encode videos very well, though, especially when paired with a mid-range GPU. For gaming, you'll get performance on par with an i5 6500. For virtualization, you'd need several users to properly leverage those CPUs.

 

[BLACKBERREST3]

I had to rethink my approach. I think you are absolutely right on this. I have a very rough idea of what I should do instead. I'll keep my current build for entertainment purposes because that is what it was built for, and I will build a workstation specifically meant for data management and storage that can also be recycled into a node later. I am deciding between the Z10PE-D16 WS and the X10DRG-Q. I am used to asus, but I am willing to try supermicro. They have the same chipset, so not much is different hardware config wise, but the biggest thing that stands out is that the X10DRG-Q uses a proprietary size which restricts the case selection. I don’t know what cases can be used with it, but I will try to find out. The other major difference is the software and features between the two. I will do more research in the meantime. The only thing I am worried about is how the storage will eventually fail. I was also looking at the iosafe 5-bay expansion chassis (eSATA) for on-site disaster prevention which looks nice. It says it only supports 40tb, but I’m wondering if I can increase it to 50tb with 5 10tb drives instead. I have done research on storage pools and it left me with a few questions. What version of ReFS is on windows 10? I would hope it’d be the latest, but I am not sure if that is only reserved for win2016. I’ll list some parts that might go in the new build.

Z10PE-D16 WS or X10DRG-Q
2x E5 2637
2x INTEL DC P4500 Series 4.0TB 2.5in*1
4u chassis with an optional backplane for hot-swap capability and a built-in power supply?
2x Crucial or Kingston 2400 4x16GB?*2
LSI 9305-24i with 4x (sas sff8643 to 4xsata)
10x 10tb wd golds*3
Iosafe 5-bay expansion (eSATA) with 5x10tb wd golds*4

1*They will be used with storage spaces with zero resiliency (raid 0) for performance
2*I am still trying to grasp how ranks and 4/8k comes into play with ram. I also cannot find any unbuffered ram at 2400Mhz with ecc. The two motherboards I was looking at don’t even support unregistered ram anyways, probably because of the C612 chipset.
3*They will be used with storage spaces as a mirror
4*I am debating this for a couple of reasons. Is there any other product like this? I would like to find one that has more bays and acts as an hba (with backplane?) and not as a raid controller like the 5-bay expansion. It would make for the best onsite backup solution compared to paying monthly or per gigabyte for cloud services.
Let’s just start from here, and we’ll see what to do next.

 

[the nerd 389]

I had to rethink my approach. I think you are absolutely right on this. I have a very rough idea of what I should do instead. I'll keep my current build for entertainment purposes because that is what it was built for, and I will build a workstation specifically meant for data management and storage that can also be recycled into a node later. I am deciding between the Z10PE-D16 WS and the X10DRG-Q. I am used to asus, but I am willing to try supermicro. They have the same chipset, so not much is different hardware config wise, but the biggest thing that stands out is that the X10DRG-Q uses a proprietary size which restricts the case selection. I don’t know what cases can be used with it, but I will try to find out.

As far as cases go, with servers and workstations, you normally get a chassis that's the same brand as the motherboard to ensure compatibility. Most of my advice has been implicitly based on this idea.

The other major difference is the software and features between the two. I will do more research in the meantime.

Supermicro software doesn't have much eye-candy. That sort of thing is unimportant in a datacenter. The software will, however, be outrageously well tested, and extremely difficult to break. It offers as many features as a datacenter would find useful.

The only thing I am worried about is how the storage will eventually fail.

This is pretty much a fact of life with this many drives. Storage spaces has an arguably better recovery process in high drive count systems, but if you ever have to send the drives to a data recovery service, expect to pay an outrageous sum. If you take a look at how storage spaces implements it's features, it's easy to see how difficult it would be to manually recover the data.

I was also looking at the iosafe 5-bay expansion chassis (eSATA) for on-site disaster prevention which looks nice. It says it only supports 40tb, but I’m wondering if I can increase it to 50tb with 5 10tb drives instead.

I wouldn't bet on it. The first rule of server design: respect the spec sheet.

I have done research on storage pools and it left me with a few questions. What version of ReFS is on windows 10? I would hope it’d be the latest, but I am not sure if that is only reserved for win2016.

The newer updates to Windows 10 support ReFS v3.2.

I’ll list some parts that might go in the new build.

Z10PE-D16 WS or X10DRG-Q
2x E5 2637
2x INTEL DC P4500 Series 4.0TB 2.5in*1
4u chassis with an optional backplane for hot-swap capability and a built-in power supply?
2x Crucial or Kingston 2400 4x16GB?*2
LSI 9305-24i with 4x (sas sff8643 to 4xsata)
10x 10tb wd golds*3
Iosafe 5-bay expansion (eSATA) with 5x10tb wd golds*4

1*They will be used with storage spaces with zero resiliency (raid 0) for performance
2*I am still trying to grasp how ranks and 4/8k comes into play with ram. I also cannot find any unbuffered ram at 2400Mhz with ecc. The two motherboards I was looking at don’t even support unregistered ram anyways, probably because of the C612 chipset.
3*They will be used with storage spaces as a mirror
4*I am debating this for a couple of reasons. Is there any other product like this? I would like to find one that has more bays and acts as an hba (with backplane?) and not as a raid controller like the 5-bay expansion. It would make for the best onsite backup solution compared to paying monthly or per gigabyte for cloud services.
Let’s just start from here, and we’ll see what to do next.

I did some checking. Intel ended support for UDIMM on the E5-26xx series. It's been a few years since I worked with that line, and I missed that change.

Regarding ranks of RAM, it's not usually that important to worry about it. A single stick of RDIMM can have up to 4 ranks on it, but unless you really need an insane amount of RAM, this shouldn't be an issue. Regarding 4-stick kits vs 8-stick kits, you ideally want all of the memory channels available to be used. That means you need one stick of RAM on each channel. Each CPU has 4 channels of RAM, for a total of 8 channels that should be populated. That means you should have 8 or 16 sticks of RAM. You don't need a lot of RAM, you just need the bandwidth. As such, you should get a relatively small size, and 8 sticks of it. You don't need 8x16GB of RAM. 8x8GB should be more than enough. If I were building this machine, I'd go for 8x4GB. I'd need to hear a very convincing argument before advising anyone to drop 128 GB of RAM into a single machine. I don't know of many situations outside of HPC builds or virtualization datacenters that could use that much RAM. Video editing builds might see a marginal improvement from the increase from 64 to 128 GB, but it's not even enough to recommend in situations that would make $15000/hour saved.

Regarding storage spaces, you should use at least two-way mirroring in a setup like this. It's a reliable method of handling this kind of setup, as far as I can tell. I've never personally used it, but based on what I know about it, it's going to be used in my next server build.

Regarding that NAS, it's not entirely necessary, and won't provide the level of protection you seem to be expecting. It's not much better than simply placing the drives in the workstation and setting them up as a separate volume. If you're getting a 4U chassis, you should be able to put 24+ drives in it via the hot-swap bays. In two-way mirroring or parity mode, you need 7 x 10 TB drives for a 50 TB block. This comes out to a total of 14 drives for the system if you want to have a main block and a backup block.

If you get a 24 bay chassis, you could run two 100 TB spaces, both with 2-way redundancy. That wouldn't leave room for hot spares, but it would get the job done.

If you wanted 2x50TB spaces, I'd recommend installing 16 drives, and configuring two pools with 7 drives each, and then assign each of the remaining drives to be hot spares. I'd also recommend keeping two drives on a shelf somewhere to replace any failing drives. Once you use a shelf spare, immediately order another one. This setup results in extremely little risk of:
A) Not having a place to recover data to.
B) Not being able to replace a drive on short notice, and
C) Drives failing faster than they can be replaced.

Regarding RAM speeds, RAM will not be the limiting factor in performance when running 8 channels. Data simply won't move that fast anywhere else in the system. You won't see any difference between 2133 and 2400.

Regarding chassis with included PSUs, it's a safe way to go as long as you stick with established brands. If you can stick to a single brand for the MB, chassis, and PSU, it makes life much easier to ensure compatibility.

 

[BLACKBERREST3]

Okay, I have a couple of questions relating to just storage now. What I want to do is;
-be able to use my own PSU (optional)
-connect more than 24 drives to the LSI 9305-24i
-have all of my drives plugged in, so when 1 drive fails in a dual parity setup, storage spaces will use that free space to rebuild itself essentially shrinking over time instead of just having a drive sit on the shelf.
-find a modular 4u chassis that fits a standard form factor (ssi ceb) that has 24-36 bays

The reality is that I don't know if any of this is possible except for the lsi part. The Ax1500i is modular by design and finding a 4u chassis without a PSU is pretty easy. The hard part of searching for the chassis is that it has to have ssi ceb form factor and a sas expander (port multiplier, multilane, i have no idea) to increase drive connection count to house 24-36 drives or have a space to install one. I don't know if storage spaces is that intelligent or not. It makes sense to have a system that uses your max (-amount for parity) storage until it wears out and then fixes itself when it starts to fail by using the remaining storage. I am also not sure if you can nest dual parity inside of a 2-way mirror with storage spaces or the other way around.

That clears things up for me with how ram works. So to get the max bandwidth while still having the ability to upgrade later, I would need only 8x16GB of DDR4 2400. It's the amount of ram that determines the price more than speed and 128GB is plenty.

 

[kanewolf]

If you want a TON of space, you should read the DIY articles for Backblaze and their storage pod -- https://www.backblaze.com/blog/open-source-data-storage-server/ That is the only reasonable way to get more than 24 drives IMO.

 

[the nerd 389]

If you want a TON of space, you should read the DIY articles for Backblaze and their storage pod -- https://www.backblaze.com/blog/open-source-data-storage-server/ That is the only reasonable way to get more than 24 drives IMO.

If storage is the only thing you're after, that's just about the best option on the market. The main disadvantage is that you don't get much compute power in that build. As a backup box, it would work wonderfully. In this case, if he/she gets a workstation platform, it's not difficult to ensure there's enough room in it for as many drives as you could want, though. Most of the Backblaze would be redundant if he/she also needs the compute power.

Now, whether or not they do is a different question. This initial setup probably doesn't. If they end up moving to Optane storage for the working space, the compute power would be necessary to keep up.

Also, it's worth mentioning that if the parts aren't going to be ordered for a few weeks, ThreadRipper should be able to do the job for quite a bit less than the Intel platform. It has enough connectivity to pull everything of with a single socket, even with Optane drives.

 

[BLACKBERREST3]

Storage capacity first, storage performance second. My preferences have changed a little going into this. I'm after reliability to. I can't really afford to lose terabytes of data. All of what you guys have said so far is good advice. I'm still trying to pinpoint "the" build that I'm after and I think I'm close.

 

[BLACKBERREST3]

On second thought, it might be easier to turn my current build into a workstation and build a separate server instead. I'll edit this post with new specs after I figure that out.

edit;

Server Core Components
E5 1660 V4 – cpu
Kraken x61 or H110i gtx – cpu cooler, have both
X10SRM-F - mobo
KVR24R17S8K4/16 – ram
2x EVGA 750w - psu
Intel Optane 32gb – OS location
LSI 9305-24i
6x LSI LSI00411 1m Internal Cable SFF8643 to x4 Sata
10x 10tb WD Golds – main storage
12x S-331 backplane

Backblaze Storage Pod 6.0 Components from backup pods
2x Wire Harnesses and Pigtails
Hard Drive Guides - QTY 120
Chassis
Server Rack Rails

Workstation Components
Z10PE-D16 – mobo
2x E5-2687W V4 - cpu
2x KVR24R17D8K4/64 – ram
2x Kracken x62 – cpu cooler
AX1500i – psu, have
GTX 970 – gpu, have
Xonar Essence STX – have
950 pro 512GB – OS install location, have
Wings PX1 – pcie nvme hba
LSI 9305-24i
2x DC P4500 4tb – workspace
PH-ES614LTG_BK - case

Notes on parts
Z10PE-D16 – I posted 2 questions in different places, but you can post them here. http://www.tomshardware.com/answers/id-3433045/z10pe-d16-ilm-mount.html
http://forum.corsair.com/v3/showthread.php?t=168333
Any cpu water cooler – I don’t know if these support narrow ILM mounting or if the Z10PE-D16 is narrow or square.
HBA cards – I posted my questions regarding bandwidth below
Cpu – I like to experiment with software and there are certain programs that leverage that many cores.
Ram – same as cpu
Wire Harnesses and Pigtails – Do I need 1, 2, or more?

I don't like how I'm locked into a micro atx form factor for the jbod, but if it works it seems like a really cost effective solution. If it doesn't, it might just be easier to get a 60 bay jbod with a tray to install the motherboard and cards.

 

[BLACKBERREST3]

Found 1 answer to my endless questions.
“Additionally, Storage Spaces can automatically repair mirror and parity spaces in which a disk fails by using dedicated disks that are reserved for replacing failed disks (hot spares), or more rapidly by using spare capacity on other disks in the pool.” https://technet.microsoft.com/en-us/library/hh831739(v=ws.11).aspx

So it is a self-healing file system like I thought and I can use all of my drives at once and not have them as just hot spares (which is also an option, idk why, longevity?). ReFS is really useful, and I can see the benefits of it, but there is something that still bugs me. Were I to lose the motherboard, OS, or the drive the OS is installed on, how would I be able to recover from that? Are the settings installed on each partition of every drive so I could connect them all to a different system? Answered my own question, but I will share with you guys anyways. https://blogs.msdn.microsoft.com/b8/2012/01/05/virtualizing-storage-for-scale-resiliency-and-efficiency/

I've been playing with numbers here and I have a few questions about theoretical and actual bandwidth in servers. Pcie gen3 x16 has a max transfer rate of around 15753.8462MB/s or 126.03Gbits/s. The storage pod 6.0 can hold 60 drives and the wd gold 10tb tops out at 249MB/s. The total transfer rate of all drives in this enclosure is 119.52Gbits/s internally. That would mean I need a gen3 16 lane hba to ensure the drives don't bottleneck. I did some research on 40/100gbe NICs and CNAs and I couldn't be more confused. Fiber connections are supposed to faster than copper, so how come I am seeing 10/40gbe fiber and 40/100/200gbe copper. Normally the excess of ports on a nic would be used to combine them for the benefits of failover and combined bandwidth up to the maximum that the pci lanes would allow, but now it seems that the slowest technology is no longer hard disks, but rather the interfaces used to connect them all to an array and other PCs.
Another thing I can't figure out are how exactly hba cards levy bandwidth. If I have a 6xsas port lsi hba rated for 8 lanes of gen3 (63.0154Gbits/s), then I should be able to connect 30 drives (59.76Gbits/s) to it. The only problem is that sata3.0 is 4.8Gbits/s and the most that I can directly wire to each drive without a port multiplier is 24 drives (6x [sas to 4xsata cables]). That means that if I tried to connect more than 2 wd golds to a 5-port multiplier, the drives will be bottlenecked. Before you even say that the drives will never reach those speeds (which I do agree with), there are some things that should be mentioned. Storage Spaces when used as a dual parity, which is the equivalent of raid 6, should see an increased read speed that scales with the number of drives you use (write speeds, not so much). If you use high end controllers or host bus adapters, then you should be able to close the gap of performance degradation. Did I miss something here or do I actually need a 100 to 200gbe nic and a pci gen3 x16 hba to get the most performance out of my storage?

 

[the nerd 389]

Found 1 answer to my endless questions.
“Additionally, Storage Spaces can automatically repair mirror and parity spaces in which a disk fails by using dedicated disks that are reserved for replacing failed disks (hot spares), or more rapidly by using spare capacity on other disks in the pool.” https://technet.microsoft.com/en-us/library/hh831739(v=ws.11).aspx

So it is a self-healing file system like I thought and I can use all of my drives at once and not have them as just hot spares (which is also an option, idk why, longevity?). ReFS is really useful, and I can see the benefits of it, but there is something that still bugs me. Were I to lose the motherboard, OS, or the drive the OS is installed on, how would I be able to recover from that? Are the settings installed on each partition of every drive so I could connect them all to a different system? Answered my own question, but I will share with you guys anyways. https://blogs.msdn.microsoft.com/b8/2012/01/05/virtualizing-storage-for-scale-resiliency-and-efficiency/

I've been playing with numbers here and I have a few questions about theoretical and actual bandwidth in servers. Pcie gen3 x16 has a max transfer rate of around 15753.8462MB/s or 126.03Gbits/s. The storage pod 6.0 can hold 60 drives and the wd gold 10tb tops out at 249MB/s. The total transfer rate of all drives in this enclosure is 119.52Gbits/s internally. That would mean I need a gen3 16 lane hba to ensure the drives don't bottleneck. I did some research on 40/100gbe NICs and CNAs and I couldn't be more confused. Fiber connections are supposed to faster than copper, so how come I am seeing 10/40gbe fiber and 40/100/200gbe copper. Normally the excess of ports on a nic would be used to combine them for the benefits of failover and combined bandwidth up to the maximum that the pci lanes would allow, but now it seems that the slowest technology is no longer hard disks, but rather the interfaces used to connect them all to an array and other PCs.
Another thing I can't figure out are how exactly hba cards levy bandwidth. If I have a 6xsas port lsi hba rated for 8 lanes of gen3 (63.0154Gbits/s), then I should be able to connect 30 drives (59.76Gbits/s) to it. The only problem is that sata3.0 is 4.8Gbits/s and the most that I can directly wire to each drive without a port multiplier is 24 drives (6x [sas to 4xsata cables]). That means that if I tried to connect more than 2 wd golds to a 5-port multiplier, the drives will be bottlenecked. Before you even say that the drives will never reach those speeds (which I do agree with), there are some things that should be mentioned. Storage Spaces when used as a dual parity, which is the equivalent of raid 6, should see an increased read speed that scales with the number of drives you use (write speeds, not so much). If you use high end controllers or host bus adapters, then you should be able to close the gap of performance degradation. Did I miss something here or do I actually need a 100 to 200gbe nic and a pci gen3 x16 hba to get the most performance out of my storage?

SAS uses (nearly) the same connector as SATA. The connector that you call SAS is actually known as SFF8643. It's a 4-port SAS/SATA connector. The HBA supports 24 SAS 3.0 ports. That said, if you want to run 60 drives in total, you should really run either two or three HBAs. I'd avoid the port multipliers if performance is your priority. This is where the added PCIe lanes come in handy. You can expand to your heart's content. It's also worth considering the HGST Ultrastar He12. It would reduce your bay requirements a bit, and as such, would make some of the standard Supermicro enclosures more feasible. I don't entirely trust the BackBlaze hardware, specifically the cooling system. Part of this is the natural skepticism of anyone who deals with servers to new things in general, but the last time I saw a server with fans that cheap, I was sending the drives out for recovery as a direct result of the fans being that cheap. If you got some Deltas for it, it could work pretty well.

Regarding the self healing properties of ReFS, they won't be equivalent to what I'm suggesting. If a drive goes down, the storage space will shrink. Having a hot spare prevents that, for starters. Second, when a drive fails without a hot spare, storage spaces will move the data onto any drives with available space. This means that throughput will effectively drop, as the files will be spread across fewer disks. When you replace the drive that failed, the files will have to be shuffled around in order to leverage the new drive. This means reduced performance for however long it takes for the new drive to be installed, and then a few hours (possibly days) as the system optimizes itself again. If you have a hot spare, you'll see reduced performance until the data can be moved onto the spare, and that's it. Once the new drive is installed, everything is as it was before you lost a drive (except the data is now on a drive in a different bay).

I'll reiterate my comments regarding the use of water cooling in a server: it shouldn't be done, even if you already have one. Stick with air coolers.

Regarding interface performance and network design, this situation would be best handled by a server that's designed to handle the bulk of the work with the data internally. This saves you from dealing with the extremely expensive high-bandwidth networking gear you'd need to fully load the storage devices. It's rather unusual for the storage devices not to be limited by LAN speeds. It should always be faster to work with data locally rather than remotely. If you end up being network-limited, you can either upgrade the network or downgrade the server.

Regarding the OS drive, I wouldn't put a server OS on a 32GB drive. You should use a 64GB drive as the bare minimum, preferably 128 GB.

Regarding the server motherboard, I'd go with one that supports all available RAM slots, as well as a few more PCIe connections if you're planning on adding more capacity down the road.

 

[BLACKBERREST3]

I was wondering about that. A micro atx server board is pretty hard to populate a 60 bay chassis with only 3 pcie slots. I would need a pcie splitter or riser card. The ppt (price per terabyte) is pretty high for the helium drives. That technology is too new. I never even thought about that! It would be so much easier to rdp into the server than to network it together or better yet combine the server and workstation in one. So I could use the workstation build for the most part, but now I have to find a decent 4u chassis which has been the crutch of my research this whole time. I'm almost out of ideas. The workstation build has enough lanes and slots to double as the main storage, I just need to find some way to connect it to a jbod chassis.

 

[the nerd 389]

Take a look at HBAs with external interfaces. That should solve the connectivity issues, but you should really have the workstation on a rack with the storage chassis. A simple 10U rack would be sufficient, and they're pretty cheap. As I've pointed out before, expect it to be loud. There's no easy way around that without sacrificing reliability.

You can use the remaining 2 slots for either a UPS or some networking gear.

 

[BLACKBERREST3]

I keep coming back to these two for the enclosure; [SC946SE1C-R1K66JBOD] and [SC946SE2C-R1K66JBOD]. The only difference is between the single/dual expander backplane. If I get 1-2 of LSI 9305-16e for the workstation, then this should work for a das. Does the dual backplane give more performance; i.e more bandwidth per backplane? The workstation may work in a rack, but I would rather have it in a tower. I will turn the jbod into a desk if it fits better that way. I figured that having a jbod chassis with its own backplane and power supply would be much simpler than doing a direct connect with no backplane. More expensive, but simpler. I chose a 60 bay to future proof it, and because with current hdd speeds (119.52-126.03gb/s), this would be ideal. I don't know what kind of bandwidth allocation goes on in this jbod, but even if it does bottleneck, this may be the best option I could find.

Edit; nevermind about the 9305-16e, there are only 4 ports which would limit the bandwidth. It would still be easy to route the cables through to the outside of the case.

 

[the nerd 389]

I keep coming back to these two for the enclosure; [SC946SE1C-R1K66JBOD] and [SC946SE2C-R1K66JBOD]. The only difference is between the single/dual expander backplane. If I get 1-2 of LSI 9305-16e for the workstation, then this should work for a das. Does the dual backplane give more performance; i.e more bandwidth per backplane? The workstation may work in a rack, but I would rather have it in a tower. I will turn the jbod into a desk if it fits better that way. I figured that having a jbod chassis with its own backplane and power supply would be much simpler than doing a direct connect with no backplane. More expensive, but simpler. I chose a 60 bay to future proof it, and because with current hdd speeds (119.52-126.03gb/s), this would be ideal. I don't know what kind of bandwidth allocation goes on in this jbod, but even if it does bottleneck, this may be the best option I could find.

Edit; nevermind about the 9305-16e, there are only 4 ports which would limit the bandwidth. It would still be easy to route the cables through to the outside of the case.

The 9305-16e isn't a bad option. That's a pretty standard solution, actually. It's not entirely accurate to say that 16 SAS 3.0 connections will limit bandwidth, though. Remember, each of those connectors is essentially 4 SAS 3.0 connections in a single cable. This will give you 4*4*12 Gb/s of theoretical bandwidth, or 192 Gb/s. It will not be the limiting factor.

Regarding those chassis, they have plenty of drive bays, to be sure, but you're currently looking for a setup that's able to handle about 25-30 drives max. Considering that for the price of those either of the ones you mention, you can get three 24-bay chassis or two 24-bay chassis and an HBA, I'd avoid either of them.

You could also get two 44-bay chassis for that price.

The only reason to go with a 60-bay top-load chassis is if you run out of rack space and it's more expensive to add a rack than it is to replace the existing servers. They're harder to service due to the top-loading drive bays, and much louder due to the added restriction of running that many drives in a single case (requires faster fans for adequate airflow).

Regarding having a server chassis as a desk, I wouldn't. They aren't designed to be run in the same room as people. They're loud. Apparently I haven't driven home how loud they are. I have a low power 2U server that burns about 200 watts when loaded up. I can hear it through two doors and down a hall quite clearly with it running the fans at around 30-40%. I would not want this in any room I spend much time in, and building it into my desk would be out of the question. If you'd like, I can go measure the sound levels for you.

 

[USAFRet]

'Loud' = something just slightly quieter than a hair dryer.
OK for 5 minutes....not something you want in your actual desk, 24/7.

 

[the nerd 389]

'Loud' = something just slightly quieter than a hair dryer.
OK for 5 minutes....not something you want in your actual desk, 24/7.

You have a server quieter than a hair dryer?

How'd you pull that off? I normally compare mine to leaf blowers.