Build Advice Can we improve on this 40TB homebrew server design?

[spacemanspiff2000]

Will some friendly folks take up this challenge? Briefly (I'll fill in any details if requested)... I'm currently running:

• 12 SATA HDDs under FreeBSD with ZFS-software-only-RAIDs:
  • 6x 4TBs in a RAID-6
  • 2x 6TB mirrored
  • 2x 8TB mirrored
  • 2x 10TB mirrored
• Intel-CPU'd motherboard with:
  • 8 SATA ports + a 4-port add-on card
  • 1x gigabit RJ45 network port
  • several USB3s normally used for attaching external HDDs for backups of the mirrors
  • VGA/DVI connectors for a simple, very old monitor (rarely on/used)
  • G2120 Ivy Bridge
  • hardware RAID not-used/not-desired in favor the software RAIDs
• OS lives on external flash drive (yeah, really ... not awesome, I know, but every time one wears out, it's time to upgrade FreeBSD anyway ... super-easy and cheap to change out)
• mostly used for:
  • Plex server (capturing/serving live TV recordings)
  • Blu-Ray/DVD backups
  • main storage for some laptops
  • home picture & video backups
  • general home file storage (finances, school, etc.)

Even though it now died on the most recent reboot, this thread is not about fixing that -- almost positive its a mobo failure -- this is about redesigning the server (or just verifying its current design).

For example, what I'm intending to do with it now is to update the mobo/CPU/RAM triad, improve the CPU for better live transcoding performance, BUT finding a mobo with 8-SATA ports for a non-obscene price is now very difficult (current mobo was bought "used" for that reason). Plus, moving this thing whenever it needs work is freaking heavy (esp. on the frontend where the drives live), and I'm not getting any younger.

So, if there are no do-able improvements, I'll just get a 6-SATA mobo and a second add-on SATA card and continue as-is.

HOWEVER, my intent here is to see if I'd be missing out on an opportunity in doing that.

Like maybe there's a good design where the drives live in a separate box outside the mobo's box? But then what/how to do that? Or maybe something with some hot-swapping of drives (or just easier access for cold-swapping)? I hear "SAS" is a thing but I don't really understand it, and if that requires changing all the drives, that's off the table. Maybe something like those things or something that's just well beyond my current knowledge.

Anyway, that's the direction I'm thinking in and my intent for this thread. What say the community here?

 

[Lutfij]

My question is, is the system/server built in a chassis? Rackmount? Regular desktop chassis? You could parse the specs of your server to help us further understand if there's any missed opportunity for you and to streamline things. In fact, you might want to look into a small form factor NAS, that should help with reducing the size of the server you're referring to then again some functionality is lost when going for an off the shelf NAS.

 

[spacemanspiff2000]

Yes, of course, good question @Lutfij. This is all presently in a tall desktop/server chassis. The Rosewill Thor V2.

When you say "parse the specs" do you mean exact names/links for each component (except probably every single HDD model is not important here)? That would be no problem, I just thought that would be "too much" at the outset. Thanks for your thoughts so far.

 

[USAFRet]

Be very very careful about your mirrored RAID things.
That data needs an actual backup. A mirror is only good for physical drive fail, not data protection.

 

[spacemanspiff2000]

Thanks, @USAFRet (and thank you for your service). Yes, I expected this would come up.

Actually, some of what's on there are backups, so those parts don't need backups. Some other parts are "files of convenience," so like if I lost all my PDF downloads of credit card statements, I could just download them again or call for older stuff. In other words, completely replaceable.

But for backing-up the other parts, I do definitely have 1, and try to have at least 2, other external drives of the same size for each mirrored pair. I know it's not conventional but it does the job: I add an external drive, make a 3-way mirror, let it sync, then drop it, and put it on the shelf. That's my snapshot in time, until the next time I rotate to that drive. This is beyond the point-in-time snapshotting features of ZFS, of course.

Beyond that, I don't know how I would be able to do backups on this in any reasonable way. Last time I did an actual "incremental backup" was back when you could put tape cartridges into PCs.

 

[USAFRet]

Ok, just checking. As long as there is more than one actual copy, not just in a mirror.
Far far too many people think RAID 1 mirror IS a backup.

My current QNAP NAS is ~65TB. Soon to be ~80TB as soon as Toshiba sends a replacement 16TB that failed.

House PCs back up to the NAS every night.
Data on the NAS (movie/music/etc) backs up to other space in the NAS. Inaccessible from the PCs, until I want to.

 

[spacemanspiff2000]

So, if this helps anyone better answer my challenge, here's exactly what I'm working with so far (with the option to replace these parts of course ... b/c that's what I'm up to here):

• case: Rosewill - THOR V2 Gaming ATX Full Tower
• mobo: ASRock B75 Pro3 (the used part that apparently catastrophically failed upon reboot)
• CPU: Intel Pentium G2120 Ivy Bridge (installed before I realized I wanted Plex to transcode)
• RAM: 16GB, 2 sticks of Patriot PGD316G1333ELK DDR3-1333 (PC3 10666) (being replaced due to mobo replacement; time for DDR4)
• monitor: Dell E198WFP 19" LCD (only matters for the video connectors)
• PS: CORSAIR HX Series HX750 Modular
• SATA card: IOCrest 4-port SATA3 PCIe2.0 x1 Controller Card
• HDDs: the mentioned 12x std sized HDDs
• misc: 2x internal HDD "cages" (I'm sorry, I see that I don't have this hard data/link handy) ... they EACH make 3 std HDDs fit into 2 upper chassis bays, each unit with its own fan in the front, molexed to the PS
• case fans:
  • 2x Rosewill 230mm on the lower-front and side (probably came with the box)
  • 2x uphere BK143 140mm on the back and bottom
  • 1x BitFenix Spectre 230mm on the top (if I have my inventory up-to-date)
• I'm sure I have some splitters in there for the fans beyond mobo capacity (or reach)

[Also, if it matters, the current failure is 6-beeps, repeat, then 3-beeps, with power/fans flowing but no response at all on the monitor.]

 

[punkncat]

I would:

consider the age of your drives and if they need replaced

consider whether you have a need for a low power system that can be used for other tasks somewhere which can act as double duty*

consider purpose built hardware such as suggested above

So, in my own case I typically have enough hardware running that I can dual purpose a system for something like HTPC/Server duty. I purchased a specific NAS unit, thinking I was doing myself a favor for backup and come to find that the format it uses is worthless to me in anything outside the enclosure. To add to that aspect, it's hot swap ability and rebuild is sketchy as hell and hasn't worked properly for me so far. It is a Synology and I absolutely would not recommend this particular 2 drive model.
I would personally use something with multiple drives and parity. (if I were to choose again knowing what I do now)

With all that said. I keep DVD storage of some critical items I don't want to lose along with storage, back up storage, AND cloud storage.
I am at a point now where I could lose any one or many of my computer and storage related devices and lose nothing of significance.

 

[spacemanspiff2000]

Thanks, @punkncat . I'm going to take those thoughts into consideration. Something else I just thought of:

Q: Would 2 Ethernet ports be of any real benefit or would it not really matter (perhaps due to a different bottleneck or just unimpressive load)?

A "stressful" use case here is probably 2 computers copying files up & down to/from this server while maybe 2-3 devices are getting served different things from its Plex (bearing in mind that Plex gets a live cable TV feed from a SiliconDust CableCard box elsewhere on the network; otherwise, the Plex server lives in the same box as the content being served).

 

[spacemanspiff2000]

I'm going to take the responses to-date to mean that I am basically on the right track and should probably just continue with my current hardware architecture, only upgrading the core "triad" that I now require.

I haven't yet seen something to the effect of "your setup is 'just OK' but what most people in-the-know who are doing this use is: <some significantly different architecture than what I have>".

So, thanks to all here for confirming my pathway. Much appreciated.