Question How to manage 400k jpg files after hard drive recovery ?

[jonathan1683]

Hello,
I was wondering if anyone has any tips I can use. I helped a friend recover their hard drive and his wife seems to catalog everything in their life videos and pictures. I kind of want to batch move every 10,000 jpgs into a sub-directory so we can manage them easier, so I would like to know if anyone knows how to handle that ?

I have a high end PC with i9-11990k, 32GB ram and a 980pro and my system really struggles handling them all, not to mention a lot of the files are corrupt and explorer sometimes crashes. I don't intend to keep working on this because the task is too much for me, but I did want to make it easier for them to do on their own and I am sure their pc cannot handle the volume of images. They will need to start viewing and deleting everything they want .

Tons of the files were recovered from like apps and temp internet files and were possibly previously deleted so a lot need to be trashed. If anyone has any ideas other than just moving them I would appreciate that too. I thought of mass delete low res or tiny file sizes not sure what else I can do to weed out the bad ones ?

 

[Nine Layer Nige]

What hardware is the images/vids stored ? (internal HDD, internal SSD, ext USB2....etc)
If the images/vids are stored on an ext hdd perhaps USB2 interface, which is going to be SLOOOOOW.

 

[Lafong]

I manage 35,000 mp3s totaling 145 gb across several hundred folders, so I run into similar issues.

A jpeg duplicate detector might be useful?

Are all of these pix currently scattered around dozens or hundreds of folders?

Is your current method to go to each individual sub-folder and then doing a simple drag and drop of all files to another drive? Big problem if there are thousands of folders that have to be done one at a time.

Are you willing to have them all land in a single directory and then manually sort them into a proper folder structure?

Or do you want any copy/move to retain the original file path?

My method for mp3s follows. You can use it for jpegs. This does NOT retain the original folder structure. All files land in a single directory of your choice. If there are any duplicate names, you'll end up with Yadayada.jpg and Yadayada (2).jpg:

Install Everything search tool from voidtools.com. Have it search the drive in question for your file extension, presumably .jpg. It will immediately give you thousands of hits. The list is sortable.

Highlight maybe 5,000 at a time and simply paste that into your chosen destination directory.

I use 5,000 at a time on mp3s. That would be about 20 GB. I have tried 10,000 and more and get choking on a less powerful system than yours.

Jpegs are typically smaller than mp3s, so maybe you can get away with pasting 10k or 20k jpegs at a time. Never tried it with jpegs.

You can also do this with 7zip, but it's tediously slow.

I think copying flat into a single directory can also be done with xxcopy. It’s strictly a command line program. I think the command is:

xxcopy source destination /sr

It’s a copy, not a move. Play with the command to confirm results.

If she has 2 files named cat.jpg in 2 different directories, you'll end up with slightly different file names in the destination folder to avoid over-writing. Is that a show-stopper?

Use the following command to rebuild the subdirectories as you might do if copying the copies back to a directory when you wanted to KEEP the old original subdirectories:

Xxcopy source destination /srr

Use xxcopy at your own risk. It's powerful, but I have little personal experience, unlike the Everything tool method.

 

[DecAns]

I thought of mass delete low res or tiny file sizes not sure what else I can do to weed out the bad ones ?

There is a plugin for TotalCommander file manager that can show jpeg file integrity:

Total Commander has "Search" function with using of plugins. It can help you to find and delete all bad jpeg files.

 

[kanewolf]

Hello,
I was wondering if anyone has any tips I can use. I helped a friend recover their hard drive and his wife seems to catalog everything in their life videos and pictures. I kind of want to batch move every 10,000 jpgs into a sub-directory so we can manage them easier, so I would like to know if anyone knows how to handle that ?

I have a high end PC with i9-11990k, 32GB ram and a 980pro and my system really struggles handling them all, not to mention a lot of the files are corrupt and explorer sometimes crashes. I don't intend to keep working on this because the task is too much for me, but I did want to make it easier for them to do on their own and I am sure their pc cannot handle the volume of images. They will need to start viewing and deleting everything they want .

Tons of the files were recovered from like apps and temp internet files and were possibly previously deleted so a lot need to be trashed. If anyone has any ideas other than just moving them I would appreciate that too. I thought of mass delete low res or tiny file sizes not sure what else I can do to weed out the bad ones ?

I would probably COPY 1000 - 5000 files to a new SSD and work on them there. That way the recovered files are still available as backup.
Then work on managing them on the new SSD.

 

[jonathan1683]

Thanks for the ideas! I definitely need to run a duplicate file finder that would help i forgot about that I use to do that with mp3s and pics before, but i forgot about it. I like the total commander idea too never knew i could see the level of corruption. I appreciate all the assistance.


Just to answer a few questions the images are on on new sandisk extreme SSD external. I tried moving them to to my 980pro , even with only 30k images it didn’t see much of a performance gain.

I have all the files in one directory which is unmanageable because windows is constantly loading with the progress bar and whenever i try to resort them it can never handle it so I think it’s windows struggling rather than hardware.

All the he pictures have a unique name like “unknown file# 848583883” and i don’t want them in one big folder because windows doesn’t like it. I was hoping to batch move them to like 100 directories. So the owner can work on like maybe a folder a day or something.

 

[Lafong]

I assume they are NOW all in one single folder containing 400,000 pictures.

And all with non-descriptive names like yayaoo3099s or whatever.

I have loaded 40,000 pix into a dupe file finder. Took the better part of an hour. Loading 400,000 may choke totally. Never tried it.

Since all pix are in one directory and all file names are useless for ID purposes, I'd guess your only move would be to one way or another copy off batches at a time to another folder and then maybe attempt to dupe detect them one by one.

I assume some human has to eyeball 400,000 pictures to properly ID them and move them to the appropriate directory. "Cat pictures"; "Dog pictures", or whatever.

I have no idea if a "corrupted" type indicator by scanning software like Total Commander has is useful or accurate enough to say "ok, let's delete that jpeg without even attempting to look at it".

On the other hand, why bother with a dupe detector unless you already know there is considerable duplication? May be a waste of time? Whoever took the pix and has been previously in charge of them would know that.

Bad situation. Someone has many hours of work to do.

 

[DecAns]

All the he pictures have a unique name like “unknown file# 848583883” and i don’t want them in one big folder because windows doesn’t like it. I was hoping to batch move them to like 100 directories.

Another idea is to move files to new folders based on the date in EXIF. As a result, there will be folders named Year (2021, 2020...) with subfolders named Month (Jun, Feb, ... or 01, 02, ...), inside which there will be corresponding jpeg files. This can be automated too.