[SOLVED] Best app, preferably open source, to help find/sort duplicate files.

[LetsPlayThisBro]

I just bought a couple new hard drives, after I install them I want to take a lot of time to sort through an old external 4TB drive another 3TB HDD with external enclosure and my current 500gb SSD containing lots of duplicate content due to backups here, backups there, backups all over the place. I want to create a file system to organize these files in a more sensible manner and get rid of all the dupes. So I'm going to be using a 500GB SSD to organize files into that new folder system I've yet to create, but I want to make sure I'm not putting dupes together again, then as I do one 500gb-ish chunk to move it onto my new 8TB backup drive as I go.

But as I transfer from one drive to the next I want to make sure, A. I don't have dupes in the new sorted folders and B, I don't have dupes on the new 8TB backup drive. Once I have all the files sorted/organized which will likely only be about 2.8-3.2 TB of data once the dupes are removed. I'll throw them onto the 4TB drive as a second external backup in case the 8TB drive dies or the 4TB drive dies I'll have it on the 8TB drive. The 3TB drive will contain copies of the files I'm actively using rather than those I simply will never delete.

This is how I was framing it in my brain anyway, if anyone else has a better method/software to make this whole process easier, preferably open source/free, I'd appreciate it. Thanks in advance.

 

[Ralston18]

Not an uncommon situation.

You may find some software that will do most of what you require.

You can also use DOS, Powershell, Python, etc.. to sort and move/copy files about.

Reference:

https://sid-500.com/2020/04/26/find-duplicate-files-with-powershell/

Maybe in a series of planned efforts that focus on one particular step or requirement. E.g., find duplicate folders and files.

Reference:

https://www.biztechreviews.com/best-duplicate-file-finder/

Likely there are multiple utility apps for any overall plan you have.

Consider that it may be easier to just set up a new partition/folder/sub-folder structure on one of your target empty drives. Then manually copy (not move) specific files to the new structure.

May take more time and effort but you (and not some utility app or script) will have full control.

Then just remove the source drive from which you copied the files and archive it. Just in case....

Key is to plan it out and not have some action cause lose of data. E.g. an older version of a data file overwrites a newer version.

Just my thoughts and suggestions. There may be others.

 

[USAFRet]

FreeFileSync does this.

FreeFileSync

Download FreeFileSync 13.5. FreeFileSync is a free open source data backup software that helps you synchronize files and folders on Windows, Linux and macOS.

freefilesync.org

Compare two folders and see what is duplicated.


For actual backups, I use Macrium Reflect.

What is your backup situation at home?

What is your backup situation at home? And if you don't do that, why not? Every single day, I read multiple threads here of "How do I get my stuff back?" or "That drive had 5 years of photos of my kids!!" Be it a dead drive, dropped phone, virus, accidental deletion, formatting the wrong...

forums.tomshardware.com