Question I need RAID advice and help to sort massive piles of data (with lots of HDDs and SSDs) ?

[Dr. Tam]

hello so i dont know exactly how to start it so i will start by saying the equimpent i have (im normal home user) and after that i will tell the story behind it..


ok so:
QNAP TR 002
SYNOLOGY DS220J
and some orico raid enclosure
HP ELITE DESK 800 G2 SFF
and a gaming pc that can support 4 hdds

now to the messy part STORAGE:

		STATUS	GB
WD​	WD GOLD​	FREE I USED TO RAID 1 THEM BUT SPLITTED THEM TO OTHERS	8000
WD​	WD GOLD​	FREE I USED TO RAID 1 THEM BUT SPLITTED THEM TO OTHERS	8000
Hitachi (HGST)​	HGST Ultrastar 7K4000​	IMPORTANT STUFF COPY FULL HDD MOST IMPORTANT DRIVE COPY OF Transcend SSD230S 4TB	4000
Seagate​	Seagate BarraCuda 4TB​	FAILING HDD	4000
TOSHIBA​	TOSHIBA MD04ACA400​	RAID 1 WINDOWS FULL HDD OF IMPORTANT STUFF inside of hp 800 g2	4000
TOSHIBA​	TOSHIBA MG04ACA400E​	RAID 1 WINDOWS FULL HDD OF IMPORTANT STUFF inside of hp 800 g2	4000
Transcend​	Transcend SSD230S 4TB (DISK 1)​	PLANING TO BREAK THE RAID 1 SSDS AND PUT THE STUFF IN HERE AND MAKING MOST IMPORTANT TOO	4000
Transcend​	Transcend SSD230S 4TB (DISK 2)​	USED TO BE RAID 1 SSD WITH IMPORTANT STUFF ON THE QNAP	4000
Transcend​	Transcend SSD230S 4TB (DISK 3)​	USED TO BE RAID 1 SSD WITH IMPORTANT STUFF ON THE QNAP	4000
WD​	WD Blue WD40EZAX 4TB​	HDD FREE	4000
WD​	WD Blue WD40EZAX 4TB​	HDD FREE	4000
WD/Transcend​	Transcend StoreJet 25H3P​	HDD free external	4000
Crucial​	MX500 CT2000MX500SSD1 3/3​	SSD FREE	2000
Crucial​	MX500 CT2000MX500SSD1 2/3​	SSD FREE	2000
Crucial​	MX500 CT2000MX500SSD1 1/3​	IMPORTANT STUFF	2000
​	​
​	​		2000
TOSHIBA​	TOSHIBA DT01ACA200​	OLD STEAM LIBRARY CAN BE FORMATED	2000
​	​		2000
WD​	WD20EFRX-68AX9N0​	hdds inside the DS220J RAID 1	2000
WD​	WD20EFRX-68AX9N0​	hdds inside the DS220J RAID 1	2000

ok the story is that i used to have that seagate 4tb drive i piled up lot of data on it music videos documents photos phone and pc backups and lot of itunes folders
this seagate disk started to show as not healty and it was always been slow so i bought 4tb toshiba disk to backup the seagate and started a chain like that.. of random piles of backups with mostly duplicates and some actual important stuff burried under.. few months ago i got hacked and i got panicked so i started to buy large disks to copy it but i saw some linus videos about raid and NAS and DAS servers and more, i ended up with too many hdds with splitted data and lot of confusion i have folders inside folders with duplicates inside folders in more duplicated folders... honestly idk what to do exactly RIGHT NOW my plan is to raid zero the 2 8TB WD GOLDS hdds and dump everything inside them and start deleting stuff the thing is idk what to do next and i know its unsafe to have raid 0, i must note that my actual most important stuff are on the TRANCSEND ssds the 4tb i used to have them raid 1 in windows but honestly idk if its needed ssd never died on me i need someone to guide me what to do because im confused... and after i sort my things out i wanna start selling the unneeded drives i just bought a lot of them out of stress and i had some pieling up over the years the the spread sheet i made here is only my big drives that im actually planning to use..
i really tried to shorten this story and now i will ask some quesstions..
(i must say im not super advanced user but i expirimented with raid 1 with external boxes and internal..)

1 since im planning to make the 8tb raid 0 to create 16tb single disk and put everything inside should i put the disks inside one of my dedicated boxes? (the orico or the QNAP) or just windows raid 0?

2 should i actually do it as raid 0 on these hdds?

3 should i keep my super important data on raid 1 ssds or it doesnt matter on ssd?

4 since i actually use my hp sff pc to orginize some files should i raid 0 the 8tb disks in to it or any others for that matter? or should i just stick it in to proper computer (the gaming pc) personally i got hacked on my gaming pc since then i use the sff for data..

5 im super confused and all over the place as you can see from this post please someone can help and give me suggestion or programs that can make my life much easier?

 

[kanewolf]

hello so i dont know exactly how to start it so i will start by saying the equimpent i have (im normal home user) and after that i will tell the story behind it..


ok so:
QNAP TR 002
SYNOLOGY DS220J
and some orico raid enclosure
HP ELITE DESK 800 G2 SFF
and a gaming pc that can support 4 hdds

now to the messy part STORAGE:

		STATUS	GB
WD​	WD GOLD​	FREE I USED TO RAID 1 THEM BUT SPLITTED THEM TO OTHERS	8000
WD​	WD GOLD​	FREE I USED TO RAID 1 THEM BUT SPLITTED THEM TO OTHERS	8000
Hitachi (HGST)​	HGST Ultrastar 7K4000​	IMPORTANT STUFF COPY FULL HDD MOST IMPORTANT DRIVE COPY OF Transcend SSD230S 4TB	4000
Seagate​	Seagate BarraCuda 4TB​	FAILING HDD	4000
TOSHIBA​	TOSHIBA MD04ACA400​	RAID 1 WINDOWS FULL HDD OF IMPORTANT STUFF inside of hp 800 g2	4000
TOSHIBA​	TOSHIBA MG04ACA400E​	RAID 1 WINDOWS FULL HDD OF IMPORTANT STUFF inside of hp 800 g2	4000
Transcend​	Transcend SSD230S 4TB (DISK 1)​	PLANING TO BREAK THE RAID 1 SSDS AND PUT THE STUFF IN HERE AND MAKING MOST IMPORTANT TOO	4000
Transcend​	Transcend SSD230S 4TB (DISK 2)​	USED TO BE RAID 1 SSD WITH IMPORTANT STUFF ON THE QNAP	4000
Transcend​	Transcend SSD230S 4TB (DISK 3)​	USED TO BE RAID 1 SSD WITH IMPORTANT STUFF ON THE QNAP	4000
WD​	WD Blue WD40EZAX 4TB​	HDD FREE	4000
WD​	WD Blue WD40EZAX 4TB​	HDD FREE	4000
WD/Transcend​	Transcend StoreJet 25H3P​	HDD free external	4000
Crucial​	MX500 CT2000MX500SSD1 3/3​	SSD FREE	2000
Crucial​	MX500 CT2000MX500SSD1 2/3​	SSD FREE	2000
Crucial​	MX500 CT2000MX500SSD1 1/3​	IMPORTANT STUFF	2000
​	​
​	​		2000
TOSHIBA​	TOSHIBA DT01ACA200​	OLD STEAM LIBRARY CAN BE FORMATED	2000
​	​		2000
WD​	WD20EFRX-68AX9N0​	hdds inside the DS220J RAID 1	2000
WD​	WD20EFRX-68AX9N0​	hdds inside the DS220J RAID 1	2000

ok the story is that i used to have that seagate 4tb drive i piled up lot of data on it music videos documents photos phone and pc backups and lot of itunes folders
this seagate disk started to show as not healty and it was always been slow so i bought 4tb toshiba disk to backup the seagate and started a chain like that.. of random piles of backups with mostly duplicates and some actual important stuff burried under.. few months ago i got hacked and i got panicked so i started to buy large disks to copy it but i saw some linus videos about raid and NAS and DAS servers and more, i ended up with too many hdds with splitted data and lot of confusion i have folders inside folders with duplicates inside folders in more duplicated folders... honestly idk what to do exactly RIGHT NOW my plan is to raid zero the 2 8TB WD GOLDS hdds and dump everything inside them and start deleting stuff the thing is idk what to do next and i know its unsafe to have raid 0, i must note that my actual most important stuff are on the TRANCSEND ssds the 4tb i used to have them raid 1 in windows but honestly idk if its needed ssd never died on me i need someone to guide me what to do because im confused... and after i sort my things out i wanna start selling the unneeded drives i just bought a lot of them out of stress and i had some pieling up over the years the the spread sheet i made here is only my big drives that im actually planning to use..
i really tried to shorten this story and now i will ask some quesstions..
(i must say im not super advanced user but i expirimented with raid 1 with external boxes and internal..)

1 since im planning to make the 8tb raid 0 to create 16tb single disk and put everything inside should i put the disks inside one of my dedicated boxes? (the orico or the QNAP) or just windows raid 0?

2 should i actually do it as raid 0 on these hdds?

3 should i keep my super important data on raid 1 ssds or it doesnt matter on ssd?

4 since i actually use my hp sff pc to orginize some files should i raid 0 the 8tb disks in to it or any others for that matter? or should i just stick it in to proper computer (the gaming pc) personally i got hacked on my gaming pc since then i use the sff for data..

5 im super confused and all over the place as you can see from this post please someone can help and give me suggestion or programs that can make my life much easier?

ANYTHING on RAID 0 has to be unimportant. RAID 0 is twice as likely to fail as a single disk. IF your data is important, then RAID0 is not recommended without a LOT of parallel storage.
SO, you need to determine how you will achieve the 3-2-1 backup solution. 3 copies on two physical devices, with one copy off-site.

 

[USAFRet]

RAID 0 - No. Not even a little bit.

RAID 1 - You need a real backup routine. RAID 1 is not that.

 

[DSzymborski]

First things first (and get the failing hard drive completely out of the equation). Get all the data that needs to be protected onto a cleaned non-RAID drive or two. Then set up a mirror using like Acronis or Macrium or Bvckup that automatically backs up everything to other non-RAID drives. Then, for the most crucial data, set up a cloud backup.

Once you've verified that you have at least two copies of everything and a proper backup plan and three copies of the important stuff, take those drives out of any PC or enclosure and format everything remaining. No janky RAID setups like it's 1993. There's nothing listed here for which a RAID makes the slightest bit of sense. This is a tangled mess of a system/network that's all mess and no benefit being derived from that mess.

 

[Dr. Tam]

Ok so after reading the comments, i have came up with temporary solution, its not the best but it might help me think clearly and focus and reduce the size of my files and duplicates manually... so i decided to create raid 0 with the 2x 8 tb wd gold disks (because i have around 4 tb accros 3 or 4 hdds .. and my prevous raid 1 8tb filled up when i did raid 1 with them so 8 tb was not enough, i want to dump everything ( copy not cut i learnd it the hard way) everything from the other drives to it, im copying using program called freefilesync, one i finish copying all of the scatterd hard disks in to this 14.5 tb raid 0 i will start deleting stuff and move important things to to an ssd, my goal is to reduce my size of unneeded stuff and to reach below 8tb of backup then i can copy the files to non raid disks and i will break the raid zero, after that i will have to probably set proper backup which i have no idea how to.. and i would actually like in depth guide for it, i am kind of noobie normal home user.. also i want to note that raid zero i did is on the qnap tr 002 i have because i might wanna copy stuff from other computers, also i trired to plug that qnap tr 002 in to my ds220j usb and it didnt work is it possible to fix it?
Also im not deleting the other drives just
Copying from them to the big storage pool of that raid 0, please share with me any thoughts about what i did

 

[Lafong]

What is your best guess as to the TOTAL size of ONE copy of "everything" you have.

4 TB?

60 TB?

 

[Misgar]

After reading all the comments you picked RAID 0.

Why?

RAID 0 is the worst possible choice for valuable data. If either drive dies, your data is toast.

Couldn't you use JBOD, or split the data across two 8TB drives, or buy a 16TB drive?

 

[Dr. Tam]

What is your best guess as to the TOTAL size of ONE copy of "everything" you have.

4 TB?

What is your best guess as to the TOTAL size of ONE copy of "everything" you have.

4 TB?

60 TB?

I would say betwin 8 to 12 tb not quite sure thats why i wanna put them all together and figure it out and delete unneeded stuff,

 

[Dr. Tam]

After reading all the comments you picked RAID 0.

Why?

RAID 0 is the worst possible choice for valuable data. If either drive dies, your data is toast.

Couldn't you use JBOD, or split the data across two 8TB drives, or buy a 16TB drive?

My most valuable data is right now sitting on single 4tb ssd which is full (i have 2 more of these 4 tb ssds and they are free), i wanna put everything together to see how big is my data i habe nothing to lose since these 8 tb disks were free and im not deleting the old data i just copy it, thats why i decided to risk it and do raid 0 obviously i have mess on the other hard drives so not quite sure what i have there but i still have the original toshiba hdds with my copied data so im good,

 

[USAFRet]

My most valuable data is right now sitting on single 4tb ssd which is full (i have 2 more of these 4 tb ssds and they are free), i wanna put everything together to see how big is my data i habe nothing to lose since these 8 tb disks were free and im not deleting the old data i just copy it, thats why i decided to risk it and do raid 0

At the end of all this copying and consolidating, and you get to a stable configuration...will you still be running a RAID 0? Or even RAID 1?

 

[Lafong]

I would say betwin 8 to 12 tb not quite sure thats why i wanna put them all together and figure it out and delete unneeded stuff,

Looks like you have about 72 TB of drive space.

Disorganization may be a much bigger problem than drive space?

Shouldn't be much of a problem if most of your drives are healthy and if you know where stuff is. Maybe you don't.

 

[Dr. Tam]

At the end of all this copying and consolidating, and you get to a stable configuration...will you still be running a RAID 0? Or even RAID 1?

Honestly i will probably break the raid 0 and split out the rest on the free 4tb ssds, after that i have no idea what to do im opend to suggestions

 

[Dr. Tam]

I wanna know how to properly backup, after i finish sorting.. because to be honest all i know is raid 1 and raid 0 and i have never done raid zero before thats why i wanted to do it now, i wanted to put it all on these big drives and move it fast to the ssds (when i finish sorting) since i heard the zero is much faster than normal hdd speed

 

[USAFRet]

I wanna know how to properly backup, after i finish sorting.. because to be honest all i know is raid 1 and raid 0 and i have never done raid zero before thats why i wanted to do it now, i wanted to put it all on these big drives and move it fast to the ssds (when i finish sorting) since i heard the zero is much faster than normal hdd speed

There are multiple different ways to backup your data.

I use Macrium Reflect. Full and Incremental Images of all my systems and their individual drives.

But read through here:

What is your backup situation at home?

What is your backup situation at home? And if you don't do that, why not? Every single day, I read multiple threads here of "How do I get my stuff back?" or "That drive had 5 years of photos of my kids!!" Be it a dead drive, dropped phone, virus, accidental deletion, formatting the wrong...

forums.tomshardware.com

Somewhat modified since I wrote that. The RAID 5 no longer exists, among a few other small changes.

 

[Ralston18]

@GalacticBear ticBear

This:

" i wanna put everything together to see how big is my data".

Not sure about the meaning there. You do not need to put everything on one disk to see " how big" the data is.

Just add up the amount of space used on the various drives.

Likely not to be an exact value.

Focus should be on safely backing up only the important files to another non-RAID drive of any sort.

Verify that the backups are recoverable and readable.

Then go back to the original drives and delete the unneeded stuff.

Speed is a lesser issue if you do not need to copy the unneeded files to begin with.

Again, no RAID of any sort.

 

[Dr. Tam]

Looks like you have about 72 TB of drive space.

Disorganization may be a much bigger problem than drive space?

Shouldn't be much of a problem if most of your drives are healthy and if you know where stuff is. Maybe you don't.

Yup, after i got hacked i got panicked and i bought many 4tb drives but i figured it confuses me

Looks like you have about 72 TB of drive space.

Disorganization may be a much bigger problem than drive space?

Shouldn't be much of a problem if most of your drives are healthy and if you know where stuff is. Maybe you don't.

Yup most of my drives are barely used and i bought them like 3 to 5 months ago, the 3x transcend 4tb sdds 2 of the wd blue 4tb and the wd 8tb golds, i also use 2 raid 1 toshibas 4tb which i copied to the new raid 0 16 tb

 

[Dr. Tam]

Btw i have few more pcs and laptops thats why i have many drives too i planned to split my data to my other computers but thats not my focus right now

 

[Dr. Tam]

@GalacticBear ticBear

This:

" i wanna put everything together to see how big is my data".

Not sure about the meaning there. You do not need to put everything on one disk to see " how big" the data is.

Just add up the amount of space used on the various drives.

Likely not to be an exact value.

Focus should be on safely backing up only the important files to another non-RAID drive of any sort.

Verify that the backups are recoverable and readable.

Then go back to the original drives and delete the unneeded stuff.

Speed is a lesser issue if you do not need to copy the unneeded files to begin with.

Again, no RAID of any sort.

I see, im not planning to keep my stuff on that raid 0 16 tb, my goal is to reduce my uneeded stuff but since i tried to do copy paste befefore and even cut paste from disk to disk i had path too long errors on windows since i have mess of folders inside folders plus music collections with long names, i dont wanna deal with it thats the main reason i wanted to put them all at one spot

 

[USAFRet]

I see, im not planning to keep my stuff on that raid 0 16 tb, my goal is to reduce my uneeded stuff but since i tried to do copy paste befefore and even cut paste from disk to disk i had path too long errors on windows since i have mess of folders inside folders plus music collections with long names, i dont wanna deal with it thats the main reason i wanted to put them all at one spot

Many people get to the point of very disorganized data.
I know I did at one point.

Fixing years worth of scattered files can be a real pain, but it needs to be done.

We're just saying doing this consolidation, with a RAID 0 in the mix, is pretty hazardous.

 

[Dr. Tam]

@Lafong said it best apparently my biggest problem is disorganization, also when i start browsing these folders i get instant confusion demotivated to sort them when i see the mess

 

[Dr. Tam]

Many people get to the point of very disorganized data.
I know I did at one point.

Fixing years worth of scattered files can be a real pain, but it needs to be done.

We're just saying doing this consolidation, with a RAID 0 in the mix, is pretty hazardous.

Yes i get it but to be honest with you part of me want it all to be gone so i wont have to think of that all the time in the back of my head, yes it has to be done i know thats why im here jn the 1st place i dont even know how or where to start or even how do i start sorting it all

 

[Ralston18]

You do not need RAID 0 or RAID 1 or any RAID at all to copy files to one target drive.

And putting all of the folders, subfolders, and files on one disk is just moving the mess around and likely will result in a bigger mess.

Focus on cleaning up one drive at a time as is.

And there are many DOS and Powershell Commands that can be used to find, copy, and otherwise manipulate files.

That aside, the real key is to take some time to look at the "mess" and plan out your housecleaning beforehand.

Really not a technical problem....

 

[Lafong]

Yes i get it but to be honest with you part of me want it all to be gone so i wont have to think of that all the time in the back of my head, yes it has to be done i know thats why im here jn the 1st place i dont even know how or where to start or even how do i start sorting it all

Pick broad categories. Maybe 8 or 10.

Maybe:

video
pictures
work-related
mp3s and music
computer related
family stuff
financial and legal related
all other

Or whatever broad categories appeal to your sense of organization.

Subdivide each of them by more categories. MP3s might be split into rock, country, blues. Pictures into family, friends, cars, animals. Whatever works for you personally.

I've got 141,000 files and 27,000 folders on a single data drive. With proper file names and a decent search engine, I can nearly always find anything within a few seconds.

Bite off a little each day. Don't get overwhelmed by it.

You will have a major problem if you are the type who is inclined to give a file name like "cool idea.doc" to a spaghetti sauce recipe. If your file and folder names are not self-descriptive, you may have to open thousands of files to find out where they should ultimately be placed.

Make a text or Word file to develop a sorting and backup plan and refine it as you get a better understanding of how to attack the problem.

 

[Dr. Tam]

Pick broad categories. Maybe 8 or 10.

Maybe:

video
pictures
work-related
mp3s and music
computer related
family stuff
financial and legal related
all other

Or whatever broad categories appeal to your sense of organization.

Subdivide each of them by more categories. MP3s might be split into rock, country, blues. Pictures into family, friends, cars, animals. Whatever works for you personally.

I've got 141,000 files and 27,000 folders on a single data drive. With proper file names and a decent search engine, I can nearly always find anything within a few seconds.

Bite off a little each day. Don't get overwhelmed by it.

You will have a major problem if you are the type who is inclined to give a file name like "cool idea.doc" to a spaghetti sauce recipe. If your file and folder names are not self-descriptive, you may have to open thousands of files to find out where they should ultimately be placed.

Make a text or Word file to develop a sorting and backup plan and refine it as you get a better understanding of how to attack the problem.

i appriciate the tips i will try today and update the thing is im music junkie and i have music everywhere most of my terabytes of data are music files (flacs wavs and itunes libreries).. i almost finished putting everything on the 14.5 tb im using around 10 tb so far

 

[Dr. Tam]

i have some stuff on my nas not important stuff (like windows iso files and stuff like that) i tried to connect my qnap DAS in to my synology ds220j nas and it said it unable to connect to the usb device is there any way to fix it?