Build Advice Can we improve on this 40TB homebrew server design?

spacemanspiff2000

Distinguished
Sep 8, 2009
27
0
18,530
0
Will some friendly folks take up this challenge? Briefly (I'll fill in any details if requested)... I'm currently running:
  • 12 SATA HDDs under FreeBSD with ZFS-software-only-RAIDs:
    • 6x 4TBs in a RAID-6
    • 2x 6TB mirrored
    • 2x 8TB mirrored
    • 2x 10TB mirrored
  • Intel-CPU'd motherboard with:
    • 8 SATA ports + a 4-port add-on card
    • 1x gigabit RJ45 network port
    • several USB3s normally used for attaching external HDDs for backups of the mirrors
    • VGA/DVI connectors for a simple, very old monitor (rarely on/used)
    • G2120 Ivy Bridge
    • hardware RAID not-used/not-desired in favor the software RAIDs
  • OS lives on external flash drive (yeah, really ... not awesome, I know, but every time one wears out, it's time to upgrade FreeBSD anyway ... super-easy and cheap to change out)
  • mostly used for:
    • Plex server (capturing/serving live TV recordings)
    • Blu-Ray/DVD backups
    • main storage for some laptops
    • home picture & video backups
    • general home file storage (finances, school, etc.)
Even though it now died on the most recent reboot, this thread is not about fixing that -- almost positive its a mobo failure -- this is about redesigning the server (or just verifying its current design).

For example, what I'm intending to do with it now is to update the mobo/CPU/RAM triad, improve the CPU for better live transcoding performance, BUT finding a mobo with 8-SATA ports for a non-obscene price is now very difficult (current mobo was bought "used" for that reason). Plus, moving this thing whenever it needs work is freaking heavy (esp. on the frontend where the drives live), and I'm not getting any younger.

So, if there are no do-able improvements, I'll just get a 6-SATA mobo and a second add-on SATA card and continue as-is.

HOWEVER, my intent here is to see if I'd be missing out on an opportunity in doing that.

Like maybe there's a good design where the drives live in a separate box outside the mobo's box? But then what/how to do that? Or maybe something with some hot-swapping of drives (or just easier access for cold-swapping)? I hear "SAS" is a thing but I don't really understand it, and if that requires changing all the drives, that's off the table. Maybe something like those things or something that's just well beyond my current knowledge.

Anyway, that's the direction I'm thinking in and my intent for this thread. What say the community here?
 

Lutfij

Titan
Moderator
My question is, is the system/server built in a chassis? Rackmount? Regular desktop chassis? You could parse the specs of your server to help us further understand if there's any missed opportunity for you and to streamline things. In fact, you might want to look into a small form factor NAS, that should help with reducing the size of the server you're referring to then again some functionality is lost when going for an off the shelf NAS.
 

spacemanspiff2000

Distinguished
Sep 8, 2009
27
0
18,530
0
Yes, of course, good question @Lutfij. This is all presently in a tall desktop/server chassis. The Rosewill Thor V2.

When you say "parse the specs" do you mean exact names/links for each component (except probably every single HDD model is not important here)? That would be no problem, I just thought that would be "too much" at the outset. Thanks for your thoughts so far.
 

spacemanspiff2000

Distinguished
Sep 8, 2009
27
0
18,530
0
Thanks, @USAFRet (and thank you for your service). Yes, I expected this would come up.

Actually, some of what's on there are backups, so those parts don't need backups. Some other parts are "files of convenience," so like if I lost all my PDF downloads of credit card statements, I could just download them again or call for older stuff. In other words, completely replaceable.

But for backing-up the other parts, I do definitely have 1, and try to have at least 2, other external drives of the same size for each mirrored pair. I know it's not conventional but it does the job: I add an external drive, make a 3-way mirror, let it sync, then drop it, and put it on the shelf. That's my snapshot in time, until the next time I rotate to that drive. This is beyond the point-in-time snapshotting features of ZFS, of course.

Beyond that, I don't know how I would be able to do backups on this in any reasonable way. Last time I did an actual "incremental backup" was back when you could put tape cartridges into PCs.
 

USAFRet

Titan
Moderator
Mar 16, 2013
153,636
10,985
175,990
24,031
Ok, just checking. As long as there is more than one actual copy, not just in a mirror.
Far far too many people think RAID 1 mirror IS a backup.

My current QNAP NAS is ~65TB. Soon to be ~80TB as soon as Toshiba sends a replacement 16TB that failed.

House PCs back up to the NAS every night.
Data on the NAS (movie/music/etc) backs up to other space in the NAS. Inaccessible from the PCs, until I want to.
 
Reactions: spacemanspiff2000

spacemanspiff2000

Distinguished
Sep 8, 2009
27
0
18,530
0
So, if this helps anyone better answer my challenge, here's exactly what I'm working with so far (with the option to replace these parts of course ... b/c that's what I'm up to here):
[Also, if it matters, the current failure is 6-beeps, repeat, then 3-beeps, with power/fans flowing but no response at all on the monitor.]
 

punkncat

Distinguished
Ambassador
I would:

consider the age of your drives and if they need replaced

consider whether you have a need for a low power system that can be used for other tasks somewhere which can act as double duty*

consider purpose built hardware such as suggested above

So, in my own case I typically have enough hardware running that I can dual purpose a system for something like HTPC/Server duty. I purchased a specific NAS unit, thinking I was doing myself a favor for backup and come to find that the format it uses is worthless to me in anything outside the enclosure. To add to that aspect, it's hot swap ability and rebuild is sketchy as hell and hasn't worked properly for me so far. It is a Synology and I absolutely would not recommend this particular 2 drive model.
I would personally use something with multiple drives and parity. (if I were to choose again knowing what I do now)

With all that said. I keep DVD storage of some critical items I don't want to lose along with storage, back up storage, AND cloud storage.
I am at a point now where I could lose any one or many of my computer and storage related devices and lose nothing of significance.
 

spacemanspiff2000

Distinguished
Sep 8, 2009
27
0
18,530
0
Thanks, @punkncat . I'm going to take those thoughts into consideration. Something else I just thought of:

Q: Would 2 Ethernet ports be of any real benefit or would it not really matter (perhaps due to a different bottleneck or just unimpressive load)?

A "stressful" use case here is probably 2 computers copying files up & down to/from this server while maybe 2-3 devices are getting served different things from its Plex (bearing in mind that Plex gets a live cable TV feed from a SiliconDust CableCard box elsewhere on the network; otherwise, the Plex server lives in the same box as the content being served).
 

spacemanspiff2000

Distinguished
Sep 8, 2009
27
0
18,530
0
I'm going to take the responses to-date to mean that I am basically on the right track and should probably just continue with my current hardware architecture, only upgrading the core "triad" that I now require.

I haven't yet seen something to the effect of "your setup is 'just OK' but what most people in-the-know who are doing this use is: <some significantly different architecture than what I have>".

So, thanks to all here for confirming my pathway. Much appreciated.
 
Last edited:

ASK THE COMMUNITY