[SOLVED] Custom RAID 6 Storage With 6 x 3.5 SATA HDD Hot Swap USB 3.1 Gen2

Toggle sidebar Toggle sidebar
ismail783

ismail783

Distinguished
Sep 17, 2008
23
0
18,510
I am very new to RAID.

As far I know RAID 6 can withstand two disk failures at one time. This is what I want.

I heard that modern RAID controllers store metadata on the HDD so we can easily replace one controller with another. I need a controller that I can swap easily without loosing data (in case the controller dies). I do not want to match the model etc or be the victim of vendor lock-In (this is the main reason I am considering a custom solution instead of a off the shelf solution).

I need 6 x 3.5 SATA hot swappable bays. It needs to have HDD status indicator like PowerEdge server (Ex. amber blinking light that says the HDD need replacement. or, alternating amber AND green to say that the HDD is about to fail) or something like that. The idea is, when I see the indicator, I just replace the HDD and rest is done automatically (rebuild the raid array and stuff like that).

The data connection from my PC to the DAS needs to be as fast as possible. I am considering USB 3.1 Gen2 (10 Gbps) or anything faster.

I am using Linux, in case this information become relevant at any point.

So, my questions are:

What casing do I need to use?

What RAID controller do I need to use?

I am looking for RAID storage (DAS), not NAS.

P.S. I am actually willing to consider off the shelf solution if that meet my requirements (can replace the RAID controller, support RAID 6, has 6-bays, has USB 3.1 Gen2, has failing or failed hdd indicator, HDDs are hot swappable).
 
Sort by date Sort by votes
There's a lot of info missing, such as use case, cost, portability.

Personally I'd use a file system such as zfs or btrfs as these support lots of data and drive redundancy option without the need for a dedicated raid controller. You can, for instance, nominate a drive as a hot spare and this will automatically replicate a failed drive.

If data resilience is a factor, I'd also look into sas drives over sata
 
Upvote 1 Downvote
anscarlett said:
There's a lot of info missing, such as use case, cost, portability.

Personally I'd use a file system such as zfs or btrfs as these support lots of data and drive redundancy option without the need for a dedicated raid controller. You can, for instance, nominate a drive as a hot spare and this will automatically replicate a failed drive.

If data resilience is a factor, I'd also look into sas drives over sata
Click to expand...
I have one. Attached to my QNAP NAS.
It is currently $245 at Amazon. $200 when I bought mine a couple years ago.
https://www.amazon.com/QNAP-TR-004-Enclosure-Attached-hardware/dp/B07K4RC7X9

Portability? I see no reason it can simply be attached to a different PC.
Yes, it hot swaps. I've tried it.
 
Upvote 0 Downvote
USAFRet said:
A QNAP TR-004 meets all your reqs, except only 4 bay vs 6.
https://www.qnap.com/en-us/product/tr-004
Click to expand...

4-bays will also work. However, I have fey queries:

  1. does it have a Linux client. I think there are software that needs to be used to manage DAS/NAS (In most cases they do not have Linux clients).
  2. can I replace the RAID controller with any modern RAID controller in case the built-in one fails?
  3. The link you provided does not say it support RAID 6. am I missing something?
 
Upvote 0 Downvote
anscarlett said:
There's a lot of info missing, such as use case, cost, portability.

Personally I'd use a file system such as zfs or btrfs as these support lots of data and drive redundancy option without the need for a dedicated raid controller. You can, for instance, nominate a drive as a hot spare and this will automatically replicate a failed drive.

If data resilience is a factor, I'd also look into sas drives over sata
Click to expand...

It does not need to be portable.

I am first considering whether there is a solution or not. If there are multiple solutions then I will consider the cost.

Let me give you my use case. There will be 2-4 disk in my Linux machine on RAID 0.
I will frequestly use rsync to sync my data from my pc to the das.

to be fair, I do not understand RAID, and I do not like what I understand.

The solution you talked about zfs or btrfs seems interesting to me. Is there any RAID 6 equivalent. I mean, even if two HDD fails, I will be able to retain my data. If such solution exist then I will not need any extra hardware for data redundancy. It will be best and cheapest solution for me.
 
Upvote 0 Downvote
ismail783 said:
It does not need to be portable.

I am first considering whether there is a solution or not. If there are multiple solutions then I will consider the cost.

Let me give you my use case. There will be 2-4 disk in my Linux machine on RAID 0.
I will frequestly use rsync to sync my data from my pc to the das.

to be fair, I do not understand RAID, and I do not like what I understand.

The solution you talked about zfs or btrfs seems interesting to me. Is there any RAID 6 equivalent. I mean, even if two HDD fails, I will be able to retain my data. If such solution exist then I will not need any extra hardware for data redundancy. It will be best and cheapest solution for me.
Click to expand...
ZFS has options for one, two(raid5 & 6) or three parity disks. Using these raid levels there is an overhead, for example when reading lots of small files. If access speed is a factor, parity based redundancy will have an effect. I don't think btrfs raid 5 or 6 is stable as yet.

You mention using rsync primarily as a one way sync, so I assume this is for backup? How often will the data be read from the DAS? How much new data will be created and sent after the initial rsync?

How valuable is your data? Is it a complete disaster if it got lost? If so, the old adage "you can never have too many backups" still holds true.

I use btrfs with nvme ssd's on my fast computer for access speed when processing data. This is backed up to a couple of NAS devices for local storage, and also backed up to cloud storage using rclone

I mention rclone, as cloud storage is often dismissed due to uncertainty about privacy. Rclone is incredibly powerful, and can remotes can be encrypted, keeping prying eyes of your data. An rclone remote can also be mounted using FUSE.
 
Upvote 1 Downvote
ismail783 said:
As far I know, NAS do not support USB (that is why it is called NAS). Please correct me if I am wrong. I am only researching for last few days.
Click to expand...
I have a 4 bay QNAP TS-453a NAS box.

Yes, it supports USB. 4x USB 3.2 ports.
And 4x gigabit ethernet
Yes, the native OS in it is Linux based. Debian, to be exact. It does all that Linuxy stuff natively. rsync, whatever.
Hotswappable.
RAID whatever you want. 0, 1, 5, 6.

Attached to that is the TR-004 I mentioned above, attached to it via USB. The OS in the NAS treats the 4 bay TR as a 4 bay extension of the main box. All seamless.

These small commercial NAS solutions are no longer just a dumb box of drives. It is a whole Linux based server, with more capabilities than you expect.
 
Upvote 0 Downvote
USAFRet said:
What is the reason behind the desire for the RAID, of whatever type?
What will all this be used for?
Click to expand...

Recently I got a heavy bill from a data recovery service provider for recovering a 4TB HDD.

They consider themselves surgeons and charge like wise. They even refer to the HDD that they will need to take the parts from as donor HDD. When I talked to them about my data privacy, they said, it is like visiting a doctor, you have to open up and show everything.

Please check https://www.linuxquestions.org/ques...s-asking-for-password-i-never-set-4175687410/

Now I am thinking if I had a good backup strategy (even if my data was backed up in four separate 4TB HDD) then the cost would be much less. So, I am considering a backup solution.

However, Earlier I had My Cloud (NAS). Data connectivity over network was so slow that I took out the HDD from that thing and put it directly inside my casing. I am still using that WD Red HDD. Very stable.

Being said that, 3-2-1 Backup Strategy is too much expensive for me.

My best option is the Backup HDD sitting inside my PC with few parity disks.

My close second best option is the Backup HDD (with few parity disks) connected to my PC via something highspeed like USB 3.1Gen2 at least. One benifit of this is, I do not have to frequently open my casing or check my HDD as those HDD will be Hot swappable, and there will be indicators that say that my HDD is failing or has failed.

The RAID option is just first line of defense. I will back up the data to another HDD even if I use RAID.
 
Last edited:
Upvote 0 Downvote
anscarlett said:
You mention using rsync primarily as a one way sync, so I assume this is for backup? How often will the data be read from the DAS? How much new data will be created and sent after the initial rsync?

How valuable is your data? Is it a complete disaster if it got lost? If so, the old adage "you can never have too many backups" still holds true.
Click to expand...

For the first go I will backup 6-8TB of data. The rest will be incremental backup. I will read from it if I loose data from my main machine.

The data is very valuable, few years of work, my precious MOOCs collection, a lot of stuff I paid for, my official documents, planning etc.
 
Upvote 0 Downvote
ismail783 said:
I am assuming that it has a RAID controller inside it. Suppose, the RAID controller fails. Can I buy any RAID controller from the market and replace the built in one and safely rebuild my RAID array and get my data back as it was?
Click to expand...
Unknown about replacing it with just any random controller, but I would guess not.

As far as getting your data back, you should have a good backup anyway. RAID or no RAID.
RAID, in any type, is not a full protection method. It is only for physical drive fail.
 
Upvote 1 Downvote
anscarlett said:
ZFS has options for one, two(raid5 & 6) or three parity disks. Using these raid levels there is an overhead, for example when reading lots of small files. If access speed is a factor, parity based redundancy will have an effect. I don't think btrfs raid 5 or 6 is stable as yet.
Click to expand...

Thank you very much for directing me towards RAIDZ.

I am actually looking at RAID-Z2 where five 4TB HDD gives 12TB.

Does QNAP or any of the providers has any solution which has
  1. RAID-Z2
  2. 5-bays
  3. USB 3.1 Gen2 or anything faster
  4. has failing or failed hdd indicator
  5. HDDs are hot swappable
 
Upvote 0 Downvote
This is really a heaping serving of bad idea. If you're primarily trying to simply protect important data, a NAS is a far better idea, and even simply having a multiple hard drive storage hub connected by USB is a far better idea. There's no information presented that indicates your data particularly requires high speed to work with. Plus, RAID only protects you from hardware data loss.

And going all-in on RAID, without another backup option, as a completely new RAID user, and with crucial data? Putting your hand on the stove is a better idea; at least you'll still have the other hand.
 
Upvote 0 Downvote
ismail783 said:
Does QNAP or any of the providers has any solution which has
  1. RAID-Z2
  2. 5-bays
  3. USB 3.1 Gen2 or anything faster
  4. has failing or failed hdd indicator
  5. HDDs are hot swappable
Click to expand...
  1. Why does it matter? You're not sure what RAID is to begin with.
  2. Yes, many. TS-563-8G, for instance. As well as 6/8/9 bay. Along with the TR-004 I mentioned above, which integrates seamlessly into the OS and overall functionality.
    3. Yes. Mine has USB 3.2
    4. Yes. Indicator lights on the front panel.
    5. Yes. I've personally tested this function.
 
Upvote 0 Downvote
DSzymborski said:
This is really a heaping serving of bad idea.
Click to expand...

You are right. At first I did not understand it, but after some research, I think hardware RAID is better than zfs raid as far my requirements are concerned.

As far zfs and qnap is concerned, I think there is a lot of qnap products that meet my last requirement (which i am not going to pursue anymore). I think QuTS hero support zfs raid (RAID Triple Parity for QuTS hero). For example:

https://www.qnap.com/en/product/tvs-h1288x

However, I am going with the first advice of @USAFRet .

Sorry for not understanding what you guys were trying to tell me from the beginning.
 
Upvote 0 Downvote
I'm still really confused as to what you're trying to do and why we're still going to back to RAID. You still haven't given a single reason for doing this that makes sense for a RAID. Now, it's your prerogative if you wish to do something needlessly complex and needlessly expensive for no actual benefit, but we want to try and give you the best advice we can.

A basic backup doesn't need to be this complex or this expensive.

Buy an additional hard-drive. After the initial backup, just have it backup the changes every night with software.

This takes my PC about 10 minutes in most cases, every morning while I'm still asleep.

Untitled.png


I pay under $10 a month to have all this backed up in the cloud as well. For 4 TB, that's $80 for the drive and $120 a year for the cloud. And I don't have to mess around with anything.

(I also have more backups than this, but this here is the minimum, no drama, and cheap)
 
Upvote 1 Downvote
DSzymborski said:
I'm still really confused as to what you're trying to do and why we're still going to back to RAID. You still haven't given a single reason for doing this that makes sense for a RAID. Now, it's your prerogative if you wish to do something needlessly complex and needlessly expensive for no actual benefit, but we want to try and give you the best advice we can.

A basic backup doesn't need to be this complex or this expensive.

Buy an additional hard-drive. After the initial backup, just have it backup the changes every night with software.
Click to expand...

Before 2000, when I was a teen, I lost my data. Then googling was not sufficient for me. Somehow I landed on tomshardware. Probably Mr. Tom helped me himself to recover my data (that is probably how I got hooked up to forums). Anyways, after that I changed my email address to gmail and lost that previous account. Then somewhere around 2008 I created another account for something similar (probably). I am saying this because I have no doubt that you try to give the best advice you can.

You actually understood my problem and gave a proper solution. You are right, I am going to buy an (or maybe two) additional hard-drive and take daily incremental backups.

as I already mentioned:

ismail783 said:
Recently I got a heavy bill from a data recovery service provider for recovering a 4TB HDD.

They consider themselves surgeons and charge like wise. They even refer to the HDD that they will need to take the parts from as donor HDD. When I talked to them about my data privacy, they said, it is like visiting a doctor, you have to open up and show everything.
Click to expand...

I was stressed out and that is why overthinking thus over-complicating the solution.
 
Last edited:
Upvote 0 Downvote
You must log in or register to reply here.

TRENDING THREADS

Latest posts

Moderators online

Share this page

COMPANY
RESOURCES
FOLLOW
Top Bottom