Question I need RAID advice and help to sort massive piles of data (with lots of HDDs and SSDs) ?

[Ralston18]

Question regarding:

"i tried to connect my qnap DAS in to my synology ds220j nas".....

Just how was that connection attempt made? What, how, etc.....? "usb device"?

More information and details needed.

 

[Dr. Tam]

i have some stuff on my nas not important stuff (like windows iso files and stuff like that) i tried to connect my qnap DAS in to my synology ds220j nas and it said it unable to connect to the usb device is there any way to fix it?

Question regarding:

"i tried to connect my qnap DAS in to my synology ds220j nas".....

Just how was that connection attempt made? What, how, etc.....? "usb device"?

More information and details needed.

ok so i have my QNAP TR-002 and i tried to connect it via usb 3 to my DS220J Nas so i can move files from the nas to the QNAP box (with the 2x 8tb drives), my DS220J has 2 usb 3 ports (which i used before with other devices and worked fine), but when i plug the qnap box it shows the device but it give me an error that its unable to be used or something like that right now im in the middle of moving files so i cant detach the device and see the error but its something like what i said.. i might update later when i can safly dissconnect the qnap from windows.

 

[Dr. Tam]

ok so another update i have made some progress i deleted some large folders and some duplicates and mostly old backups that i probably dont need (it seems like i have full copies of some old hardrives on my hard drives if it makes sense)

ok so i made a plan its not complete one (i need help to complete it) but i will share about my progress

step 1 was making big harddrive sotrage pool so i can copy everything inside which will help me think clearly and do it fast without wasting too much time (raid 0) i just copied everything using freefilesync (disk by disk) becauese thats the program i know, (since i had long folder names problems on windows explorer and i dont trust it anymore) i dont think i lost any data by doing so, (since my data remained on their original locations.)

step 2

i have have succsesfuly copied all disks to the "big one" seen that im not reaching my full capacity of the 2 disks combind (14.5 tb) so thats good (i have reached around like 10 to 12 as i suspected with all duplicates and old backups combined and every hdd and ssd that i had), i started to use a lot of EVERYTHING SEARCH + FREEFILESYNC programs and cleaned some folders i decided to delete my 4tb important files folder from the raid zero, since i have the original importand stuff 4tb ssd anyway.. (and its duplicated to another ssd even) so im coverd here.. since i removed 4 tb from this RAID 0 BOX and cleaned big useless folders right now im sitting on free 12 tb and i wanna start breaking the raid 0. (i also have nvme with 1 tb that i Cloned to crucial mx500 2tb ssd so i removed it from the RAID0 and as we speak i remove some more.

step 3 breaking raids
so now im on step 3 "breaking raids" i wanna break the raid zero that i made and move the data to diffrent disks and have more permanent solution. i have 2 fairly new WD BLUES 4TB 5400 RPMS and 2 TOSHIBAS 7200 RPMS 4TB both of them not new (one one that i replaced my seagate with, and one that i bought off aliexpress i think it had his hours and power on time resetted if its possible so im not sure about its state it seems fine tho) but they almost same model, i wanna move my stuff from the 8tb x2 raid zero to either one or 2 of these and use maybe the rest for backup? this is where my confusion starts im kind of stuck here so right now im stuck at step 3,

many people here advised me against raid at all but i actually dont understand the reason why? im kind of noob at this subject i mean all i know is raid 1 can protect you from HARD DRIVE FAIL so this is what i used before i posted here and it actually saved me once.. i would like somebody to explain to me why not? and whats the alternatives since i dont wanna manually copy paste an hdd for 10 hours every day to diffrent hdds im pretty sure there are programs for it but im slow learner and i dont wanna mess up im not pro at these things but since i got hacked i started to watch some videos about raid and it seemed to be the right thing to do, its important to say i was stricktly using raid 1 (mostly on microsoft disk managment)


step 4 will be to just sort the rest of the remaining files


step 5 will be to achive a permenant stable backup and easy to use and maintain

step 6 (this will probably my ultimate step and goal) just make it all work and orginized and clean.. and have multible copies on external hdds maybe

 

[Dr. Tam]

and one more think i would like to know how to break raid 1 in windows without losing data on one of the 2 disks that are mirrored

 

[Ralston18]

The QNAP TR200 is, to me anyway, not a true NAS. It is a storage unit using USB to connect to a host computer.

Thus sharable on a network only if the host computer is on.

Versus network storage (DS220J) that is directly connected to the network via an Eithernet connection and has no dependency on a host computer.

RAID may still have some place in certain environments. However, the original concept has been overtaken by technology and operational practices.

Recovering via a backup being faster and easier than recovering a RAID configuration/level of any sort as I understand it. Likely there are exceptions but generally not applicable to or viable for home purposes.

FYI:

https://www.techtarget.com/searchst... and the data,longer to rebuild failed drives.

Unfortunately, you are dealing with multiple problems while trying to get the folders and files sorted out and reorganized.

As for "breaking RAIDs", I will defer to those who have worked in RAID environments and may be able to offer specific suggestions.

Overall, as I view the situation, about all you can do/continual to do is to grind through the process.

 

[USAFRet]

The QNAP TR200 is, to me anyway, not a true NAS. It is a storage unit using USB to connect to a host computer.

Right. I have the TR-004.
DAS, not NAS.

 

[DSzymborski]

RAID is not a data protection solution. It's a data *availability* solution, which is mostly a very different thing. A proper, uncomplicated backup is what you use to protect your data. RAIDs protect mainly against one kind of data loss, hard drive failure, and create additional problems in return for the one way they protect things. Backup plans are simple tasks that need simple, dependable, solutions.

Is your data a large amount of highly sequential data, in which waiting a couple extra hours to restore a backup could cost you hundreds or thousands of dollars in productivity, like editing 4K video footage or doing high end-simulation work? Are these PCs the server for a storefront in which you could lose thousands of dollars of sales? Do you have a popular YouTube channel in which a large part of the audience is 22-year-olds who find that type of usually pointless stuff cool because it's complex? Do you care about numbers in a benchmark more than real-world performance? Are you a fan of losing data in creative ways?

Those are the main scenarios for a RAID. Not backing up stuff or listening to music files off your server or accessing your old photos or almost any consumer task.

 

[Lafong]

all i know is raid 1 can protect you from HARD DRIVE FAIL ..................i dont wanna manually copy paste an hdd for 10 hours every day to diffrent hdds im pretty sure there are programs for it

There are non-RAID ways of protecting from hard drive fail.........that are much less complex and more suitable for your apparent situation.

Yes, there are "programs for it" that would NOT require you to copy and paste for 10 hours a day. You need to get acquainted with those programs.

Eventually.

But for now....continue with your present idea of getting 1 copy of all data files onto an ordinary non-RAID disk or disks. You may not have any single disk that is large enough to hold everything.

After that's done, you can devise a backup plan to protect your data.

Sounds like you might end up with a lot of lower capacity HDs you might be willing to sell and replace with HDs of much larger capacity to hold your backups. I see HDs of at least 20 TB on market, but I don't know your budget situation.

 

[DSzymborski]

Yeah, as Lafong notes, you don't have to spend 10 hours a day updating. Pretty much every good software supports delta copying, which checks the source and backup and only changes things that have changed since the previous backup. Unless I've done a lot of stuff, my PC rarely takes more than 10 minutes in the middle of the night every morning to back up all files on multiple servers (and one of the servers also sends it to the cloud).

 

[Dr. Tam]

Ok guys so i have a small update, i managed to clean most of my clutter from this raid 0 (8tb x2) i moved 85% of my important things to my important stuff ssd 4tb (its standing on around 3.2 tb of usage now) and left some files i need to compare later on this raid 0, also i kept it all under 2 tb, i was comfterable enough with this size so i cloned this raid 0 in to an empty 4tb ssd, and i plan to break the raid 0. And move forward.

So after reading what i just said i decided to make everything easier to read i will start giving letters to HDDS for everyone to understand the situation better

"Important stuff ssd" will be called "A" (TRANCSEND 230S 4TB)

The "raid 0 14tb " will be called "W"
(WD GOLD 2X 8TB)

the cloned ssd of "W " will be called "B"
(2nd TRANCSEND 230S 4TB)

The original untouched raid 1 that i copied before to the "W" DRIVE on my pc has 2 toshiba drives will be called "T"
(I forgot their model but they 4tb 7200RPM almost full) i plan to split the raid 1 and format one

And to my other 2 wd disks (wd blue 4tb 5400 rpm 2x) one of them will be called "K" and one will be called "I"

I also have 1 hitachi 4tb hdd 7200 that i have old copy of A that i use for backup. I will call it "H"

OK so now that i added letters to my drives my explanation might be easier to understand now , maybe i can explain the situation better..

 

[Dr. Tam]

With that out of the way, right now i have "A" with almost 3.2 tb of usage "B" with around 1.5 tb these are my important hard drives

I plan to break "W" and "T" from their raids

On "K" and "H" and "T" I have old copies of my data before i sorted it

Disk "i" is empty i planned to put my "T" files inside of it and maybe keep one raid 1 active and empty just in case.. but im open for ideas

(And i forgot to mention my Synology nas has 2x WD RED 2TB one of them starting to die i think and my nas has around 1 tb of Usage )

I would like someone to guide me what to do next exactly step by step, because im a little confused still as you can see..

 

[Lafong]

I'm quite confused by your organization and drive letters too, BUT.............

Is it true that K, H, and T combined contain 100% of your UNSORTED data files that you care about and want to preserve? And that only T is a RAID drive?

If yes, do the combined contents of K, H, and T exist in SORTED form on any of your other drives? If yes, which?

If yes, what is the combined space occupied on all 3 of those drives?

If no, then I don't follow your description of them as:

"On "K" and "H" and "T" I have old copies of my data before i sorted it".

I'd think your first goal would be to get ALL of your data in SORTED form on NON-RAID drives. Then worry about the next step. But I don't quite know where you are in that process. Maybe you have some other first goal.

 

[Dr. Tam]

Ye im sorry for the confusion i confused myself too this whole situation is a mess so its hard to explain but i will tell you, basicly K H and T are the data before i sorted it these are the orginal drives which i pulled all of the files from and dumped them on W i kept them as they were only for back up, W (and B are now the same) is the result of cleaning all of these drives

 

[Dr. Tam]

i kept these drives just in case i actually "cleand" something important off my W, but now since i cloned W in to ssd (B) i have no use for the raid 0 anymore i guess so just now i had an idea to just dump all of K T and H in to single WD gold 8 TB and make it as one drive for "cold backup" and format all of the others since im wasting a lot of disks for no reason, or should i use these 2 WD golds for something else? im open to ideas

 

[Dr. Tam]

my goal is to achive order organization and i guess sort of proper backup.. for normal home user.. maybe i will even sell some drives

 

[Lafong]

Ye im sorry for the confusion i confused myself too this whole situation is a mess so its hard to explain but i will tell you, basicly K H and T are the data before i sorted it these are the orginal drives which i pulled all of the files from and dumped them on W i kept them as they were only for back up, W (and B are now the same) is the result of cleaning all of these drives

I guess you mean that EVERYTHING is on K plus H plus T and that same stuff is also on W.

And that W and T are RAID drives.

Can you answer directly any of the questions in post 37?

This also confuses me: You say W contains "everything" and that B will soon contain the same as W. You say B is a 4 TB drive.

That implies that all of your data will fit on a 4 TB drive. In your first post, you say you have 18 drives totaling over 70 TB.

It certainly doesn't appear that you have a lack of storage.

I'm nearly certain there is a lot we don't understand or you haven't adequately explained.

 

[Dr. Tam]

Is it true that K, H, and T combined contain 100% of your UNSORTED data files that you care about and want to preserve? And that only T is a RAID drive?

yes

If yes, do the combined contents of K, H, and T exist in SORTED form on any of your other drives? If yes, which?

yes kind of.. see most of my unsorted files, were backups of old hard drives which i dont need and i forgot about and somehow i created another copy of the same onces.. if you understand what im saying.. so i just deleted both copies since i didnt need them, when i SORTED the files in to W i deleted the uneeded stuff and moved the important things betiwn A and W and to my NAS so i would say i do have my important things ready.

If yes, what is the combined space occupied on all 3 of those drives?

i will try to calculate it and send screenshot of "my pc" folder to make it easier to understand

If no, then I don't follow your description of them as:

"On "K" and "H" and "T" I have old copies of my data before i sorted it".

Ye these are the original disk before i made this post. K is copy of early version of A before i moved it to W (when i made this post) I saved it for backup. H is the hitatch disk, its even older copy of A but i found other partition of old copy of W in the same disk. T is backup of my old dying seagate drive in raid 1

I'd think your first goal would be to get ALL of your data in SORTED form on NON-RAID drives. Then worry about the next step. But I don't quite know where you are in that process. Maybe you have some other first goal.

ye thats whare i started and im still on my journy.. im not at the point when i can say that im fully satisfied but i can tell you one thing having the important data on "A" SSD makes me half satisifed since "A" mostly orginized and having the W data on other ssd feels more safe than having it on HDD i know i need to keep sorting but i wanna have like automatic backup of some kind (like file history or something) that i can recover some files if i delete something important while sorting.. thats kind of my goal at the moment

 

[Dr. Tam]

I guess you mean that EVERYTHING is on K plus H plus T and that same stuff is also on W.

yes and no since i deleted unneeded stuff and i moved stuff to A aswell

And that W and T are RAID drives.

Yes W is raid 0 that i can format since its copied to ssd now (B). and T is raid 1 backup.

Can you answer directly any of the questions in post 37?

ye i did sorry about it i didnt know how to reply to individual quesstion before

This also confuses me: You say W contains "everything" and that B will soon contain the same as W. You say B is a 4 TB drive.

yes W and B are basicly the same one now i can format and break the raid 0 now on W. i know its 4tb but i made my W mess to be smaller then 2 tb so it fits to the 4tb ssd i deleted a lot of things and moved some files from W in to A and to another 2TB ssd that i carry with me on the go

That implies that all of your data will fit on a 4 TB drive. In your first post, you say you have 18 drives totaling over 70 TB.
Yes and no my A drive is almost full But yes my W can fit in to 4 TB right now

It certainly doesn't appear that you have a lack of storage.

No i just have "too much" of it in small amounts also these drives i listed here these are my big drives i also have some nvmes and smaller 1tb drives and 500 gb and so on but im not speaking about them right now since they are empty and i dont want to add more confusion

I'm nearly certain there is a lot we don't understand or you haven't adequately explained.

yea kind of but we are getting there im sorry for the mess and i appriciate you for the help and everyone else too.. i will answear any quesstion to the best of my ability..

 

[Lafong]

I asked:

"do the combined contents of K, H, and T exist in SORTED form on any of your other drives? If yes, which?"

You answered: "yes, sort of".

Keep going until the answer is an unqualified "yes". Not "sort of".

When done with that, let us know how many TB that adds up to.....ALL OF IT, one copy of each file you want to keep; sorted properly.

 

[Dr. Tam]

I took a screenshot of my disks situation how can i upload it? its not letting me

 

[Dr. Tam]

I asked:

"do the combined contents of K, H, and T exist in SORTED form on any of your other drives? If yes, which?"

You answered: "yes, sort of".

Keep going until the answer is an unqualified "yes". Not "sort of".

When done with that, let us know how many TB that adds up to.....ALL OF IT, one copy of each file you want to keep; sorted properly.

i think yes to be honest i can never be 100% sure but i have my important things pulled out from the mess i mean lets say if for example right now if will lose these drives data i wouldnt care because i feel confidant i have my things that i need saved where i needed. (obviously maybe there are files i missed) but so far im not worried these drives i keep as "just in case"

 

[Lafong]

I took a screenshot of my disks situation how can i upload it? its not letting me

you should be able to upload the image to either of the sites below and then post that link in this thread.

Your pix may not be better than a description, but we'll see.

Postimages — free image hosting / image upload

Provides free image upload and hosting integration for forums. Free picture hosting and photo sharing for websites and blogs.

postimages.org

or

imgur.com

 

[Dr. Tam]

you should be able to upload the image to either of the sites below and then post that link in this thread.

Your pix may not be better than a description, but we'll see.

Postimages — free image hosting / image upload

Provides free image upload and hosting integration for forums. Free picture hosting and photo sharing for websites and blogs.

postimages.org

or

imgur.com

thanks, here we go :

Untitled — Postimages

postimg.cc

( My HDDS and SSDS having custom icons so it might be easier to understand)

 

[Lafong]

Untitled — Postimages

postimg.cc

I see 8 drives.

A B C H J K T W

Which of those 8 contain ALL of your data in sorted form? Name them.

Do those drives contain ONLY your data in sorted form and nothing else?

 

[Dr. Tam]

I see 8 drives.

A B C H J K T W

Which of those 8 contain ALL of your data in sorted form? Name them.

Do those drives contain ONLY your data in sorted form and nothing else?

A is mostly orginized out of all of them. B and W are the same and they like 50% orginized. i have external ssd which is sorted too i copied stuff in to from the mess that i had in to it too.. and my NAS is sorted aswell