Question Recommendations for a free duplicate file finder

[Pimpom]

I'm in the process of re-organizing and trimming my data in preparation for moving to a new system with all-new all-SSD storage. It will run on Windows 10.

Over the past 25 years or so, I've accumulated a lot of files backed up by dumping the contents of a folder to a backup folder marked with the date of dumping, e.g. "Desktop 14Jul12", "Desktop 23Mar17", and so on. Many such dumps contain identical and near-identical files in them. I never got around to sorting them out and want to do so now.

It's the near-identical files that I'm most concerned about. For example, there are Word files and photos that were modified after one dump and later saved under the same name in a later dump. Some files must have been carried over through more than half a dozen successive dumps.

I know there are many duplicate file finders out there but have never used any of them. Can you recommend a free one that's easy to use, fast and effective in distinguishing files of the same name but with altered contents?

 

[Lafong]

Are all of these files on 1 drive? Both the one you'd want to keep and the ones you'd want to delete.

Or on many drives?

Do the dupes always have the same name as the original? Or is it possible that 454.doc and 3x44e.doc have the exact same contents?

If 445.doc in folder 1 has different contents than 445.doc in folder 2, how would you know which you want to keep WITHOUT opening both?

Put another way, can you identify which files you want to delete by file name ALONE?

I might have some ideas for you, but need a better understanding.

There are some decent JPG dupe finders that will show you "near identical" pictures side by side in thumbnail versions and let you choose which to delete then and there, based on your visual examination of the thumbnails. But if you had 20,000 pictures to compare it can be tedious...you could do maybe 10 a minute by eye. Or would you be willing to delete WITHOUT looking at the thumbnails at all...purely by file name?

Are these files a variety of file types....Word, mp3, jpg, txt, mp4, Excel, etc?

 

[Pimpom]

Are all of these files on 1 drive? Both the one you'd want to keep and the ones you'd want to delete.

Or on many drives?

On the same physical HDD. Some are in the same partition, some in different partitions.

Do the dupes always have the same name as the original?
Or is it possible that 454.doc and 3x44e.doc have the exact same contents?

They will have the same filename in most cases. If they had different names, I wouldn't mind keeping both. There will be only a few, if any, such duplicates.

If 445.doc in folder 1 has different contents than 445.doc in folder 2, how would you know which you want to keep WITHOUT opening both?

Put another way, can you identify which files you want to delete by file name ALONE?

That's what the duplicate finder is for. A good one is supposed to be able to detect even slight differences of content and hash.

I might have some ideas for you, but need a better understanding.

There are some decent JPG dupe finders that will show you "near identical" pictures side by side in thumbnail versions and let you choose which to delete then and there, based on your visual examination of the thumbnails. But if you had 20,000 pictures to compare it can be tedious...you could do maybe 10 a minute by eye. Or would you be willing to delete WITHOUT looking at the thumbnails at all...purely by file name?

I don't need any clever software for that.

Are these files a variety of file types....Word, mp3, jpg, txt, mp4, Excel, etc?

Yep.

 

[Lafong]

These might be useful to one extent or other:

WinMerge; file and folder comparison tool; I used it successfully 10 years ago.

DupeGuru; haven't used it, seen it recommended.

FastFileDuplicateFinder.exe; I used this successfully several years ago; said to work on most file types, not just images.

 

[punkncat]

In my own experience have had these dupe eliminators absolutely ruin source files and originals. Where it is an insanely long process, you best bet is to find the most intact files/folder that you have and manually fix it. Delete all the others and copy/pasta the hand corrected one back into the other locations.

In particular this was a major issue for me after accidentally having Media Player copy and save anything it played. I ended up with a mess that I still discover ruined files or bad copies many years later. These programs have no point of reference other than file name and don't know if it is keeping the good one or not.

To the point, make a backup (even of the 'messed up' files) before setting any of these apps loose on the "fix".

 

[Pimpom]

These might be useful to one extent or other:

WinMerge; file and folder comparison tool; I used it successfully 10 years ago.

I've seen it mentioned somewhere else too. At the moment I have no idea how it goes about doing what it does. If it's about minutely laying out similar files for comparison by the user, it's not for me. I have simply too many files and folders to sort out.

DupeGuru; haven't used it, seen it recommended.

I tried it out last night. At first I was put off by the 15 duplicate files limit of the free version. Later I thought it must mean 15 duplicate copies of the same file. So I downloaded and tried it. Found that it indeed means 15 different files, each with its own duplicate. Not 15 copies of the same file. No use for me.

FastFileDuplicateFinder.exe; I used this successfully several years ago; said to work on most file types, not just images.

I'll check it out later.

 

[Pimpom]

In my own experience have had these dupe eliminators absolutely ruin source files and originals.

That's what it looks like. I've tested a couple of them but didn't blindly tell them to delete duplicate copies.
Turned out that many identical files can exist in different folders for a legitimate reason. For example, I scanned a folder that contains more than a dozen sub-folders, each of which contains my collection of motherboard drivers. The scan found more than 13,000 duplicates of various types like dll, exe, inf, jpg, gif, cab, dat, msi, sys, etc. etc. totaling more than 6GB. If I had blindly deleted those duplicates, I'd be left with a collection of unusable drivers.

Where it is an insanely long process, you best bet is to find the most intact files/folder that you have and manually fix it. Delete all the others and copy/pasta the hand corrected one back into the other locations.

It looks more and more like doing it manually is the only realistic option.

 

[punkncat]

That's what it looks like. I've tested a couple of them but didn't blindly tell them to delete duplicate copies.
Turned out that many identical files can exist in different folders for a legitimate reason. For example, I scanned a folder that contains more than a dozen sub-folders, each of which contains my collection of motherboard drivers. The scan found more than 13,000 duplicates of various types like dll, exe, inf, jpg, gif, cab, dat, msi, sys, etc. etc. totaling more than 6GB. If I had blindly deleted those duplicates, I'd be left with a collection of unusable drivers.

It looks more and more like doing it manually is the only realistic option.

Of note, I would NEVER set one of these loose on OS/support folders or files, or root of a drive. That is a recipe for disaster. I am actually surprised that the program itself would not have warned you not to do that.

 

[Eximo]

I used a tool called AntiTwin once. Seemed to produce decent results. I think the reason it stood out it would also compare contents of files. If they were identical, but had different names it would flag them for you.

I used it to go through an icon set for an 'online' storefront. Had a team of about ten people gather icons for an application library, and ended up with many duplicates.

 

[Lafong]

I've seen it mentioned somewhere else too. At the moment I have no idea how it goes about doing what it does. If it's about minutely laying out similar files for comparison by the user, it's not for me. I have simply too many files and folders to sort out.

I tried it out last night. At first I was put off by the 15 duplicate files limit of the free version. Later I thought it must mean 15 duplicate copies of the same file. So I downloaded and tried it. Found that it indeed means 15 different files, each with its own duplicate. Not 15 copies of the same file. No use for me.

I'll check it out later.

I've seen it mentioned somewhere else too. At the moment I have no idea how it goes about doing what it does. If it's about minutely laying out similar files for comparison by the user, it's not for me. I have simply too many files and folders to sort out.

I tried it out last night. At first I was put off by the 15 duplicate files limit of the free version. Later I thought it must mean 15 duplicate copies of the same file. So I downloaded and tried it. Found that it indeed means 15 different files, each with its own duplicate. Not 15 copies of the same file. No use for me.

I'll check it out later.

Re Winmerge..........

It's been over 10 years, but my recollection is that it was quite useful in comparison of 2 folder trees. If they differ and what file or folder causes them to differ. I can't recall why I needed that capability at that time.

But I'm not sure how it would be in searching for "duplicate files" per se.