• 0

Software to detect duplicates, with a difference?


Question

So to cut out a load of boring waffle for you, I need software to detect duplicate files.

 

I've tried Auslogics (as it was free) and it seemed to work well - for 100% identical files.

 

However, I have a ton of files that are 99% identical. By that I mean I have files that are basically the same but they have a -1 at the end of the file name. The rest of the file name is the same, the file size is the same, if they're a video then the duration is the same - but one will have a -1.

 

So we're talking about a file like: mycat.jpg and then another file called mycat-1.jpg ... where they're the exact same picture.

 

Is there such a software for Windows that can detect duplicates in this manner? I swear I must have many GB I could free up. I've already done 10GB off of 1 small subfolder.

8 answers to this question

Recommended Posts

  • 0

I used HashMyFiles to scan MD5 hashes (other hash types selectable) in various directories to scan for duplicate photos. Worked great. Name of file was irrelevant. You can open/scan multiple directories simultaneously. Duplicate hashes are handily color coded. Just picked which one to delete and done.

 

https://www.nirsoft.net/utils/hash_my_files.html

 

 

HashMyFiles.png

Edited by JustGeorge
  • Like 3
  • 0

If this is specific to digital camera photo's, check out https://www.advancedrenamer.com/

 

(Free for personal use)

 

It plugins to exif command line, and you can rename files to be Camera Model, Shutter Count (provided the camera embeds the the info).

 

I do like JustGeorge suggestion and taking a look at that myself for my various back up archive files.

  • 0

Thanks for the suggestions guys.

 

Turns out the software I was using, Auslogics, which is also free, does exactly what I need.

 

Issue was - I just didn't know how to use it properly at the point of asking.

 

Got it sorted now. Scanned 2 folders already & 10GB of duplicates in each for a 20GB free up.

  • 0

I used AllDup portable for video and images, but it isn't great, needs multiple passes to be able to pick out all dupes.

 

I will have to give a few of the suggestions in this topic a go.

  • 0
  On 23/09/2022 at 16:37, Technique said:

So to cut out a load of boring waffle for you, I need software to detect duplicate files.

 

I've tried Auslogics (as it was free) and it seemed to work well - for 100% identical files.

 

However, I have a ton of files that are 99% identical. By that I mean I have files that are basically the same but they have a -1 at the end of the file name. The rest of the file name is the same, the file size is the same, if they're a video then the duration is the same - but one will have a -1.

 

So we're talking about a file like: mycat.jpg and then another file called mycat-1.jpg ... where they're the exact same picture.

 

Is there such a software for Windows that can detect duplicates in this manner? I swear I must have many GB I could free up. I've already done 10GB off of 1 small subfolder.

Expand  

If the files are the same, the filename shouldn't matter because filenames don't affect checksums, so you would think Auslogics would detect it as being a duplicate anyway.

 

Just saw your follow-up comment, glad you got it sorted, 🙂

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Recently Browsing   0 members

    • No registered users viewing this page.
  • Posts

    • I don't think RTSS can show the real framerate. In fact, this is the first time I have ever heard about anything showing it.
    • Yeah, Deck's overlay has CPU temps. Maybe they don't want to rely on external dlls to provide CPU temperature graphs on all machines? From my experience with Rivatuner it can be annoying to select the correct CPU temperature sensor.
    • Why would I be praising Microsoft? You do realize I'm well aware the Gamebar is a stinking pile of crap and has been for years, right? Or are you one of those "if you don't critique something at every turn you area clearly a fanboy" people? Yes, MangoHUD is built into SteamOS's Big Picture mode, and up until a few weeks ago, SteamOS was exclusive to... the Steam Deck... for 3 years. Yet you were talking about it as if you didn't need to set it up yourself, like it was built into other distributions, which it wasn't. So if we ignore the real/fake framerate feature, it's really not much different than me setting up my Rivatuner/HWInfo overlay, is it? Valve had the option of building MangoHUD into Steam, like they are doing it now, for 3 years. So let's not pretend Windows hasn't had a basic way to display your framerate built in, keeping well aware how crap it actually is. And I'm not even counting your GPU driver company, because Nvidia, AMD and Intel have their own (somehow even worse than GameBar) overlays. And of course they won't tell you fake/real framerate, why would Nvidia nor AMD want to tell you? That is absolutely a +1 on Valve.
    • As much as I like the push towards Wayland (is Gnome still bickering about window decorations and "you are doing it wrong"ing rest of the industry?) but as an admin, the fact that the FreeRDP (because VNC is dead) implementation is not functional yet is really annoying, as that is what RHEL10, and Debian13, will be stuck with for the next 3-8 years.
    • qBittorrent 5.1.1 by Razvan Serea The qBittorrent project aims to provide a Free Software alternative to µtorrent. qBittorrent is an advanced and multi-platform BitTorrent client with a nice user interface as well as a Web UI for remote control and an integrated search engine. qBittorrent aims to meet the needs of most users while using as little CPU and memory as possible. qBittorrent is a truly Open Source project, and as such, anyone can and should contribute to it. qBittorrent features: Polished µTorrent-like User Interface Well-integrated and extensible Search Engine Simultaneous search in most famous BitTorrent search sites Per-category-specific search requests (e.g. Books, Music, Movies) All Bittorrent extensions DHT, Peer Exchange, Full encryption, Magnet/BitComet URIs, ... Remote control through a Web user interface Nearly identical to the regular UI, all in Ajax Advanced control over trackers, peers and torrents Torrents queueing and prioritizing Torrent content selection and prioritizing UPnP / NAT-PMP port forwarding support Available in ~25 languages (Unicode support) Torrent creation tool Advanced RSS support with download filters (inc. regex) Bandwidth scheduler IP Filtering (eMule and PeerGuardian compatible) IPv6 compliant Available on most platforms: Linux, Mac OS X, Windows, OS/2, FreeBSD qBittorrent 5.1.1 changelog: BUGFIX: Don't interpret wildcard pattern as filepath globbing (glassez) BUGFIX: Fix appearance of search history length spinbox (glassez) BUGFIX: Remove dubious seeding time max value (glassez) BUGFIX: Fix ratio handling (glassez) BUGFIX: Fix compilation with Qt 6.6.0 (glassez) WEBUI: Make General tab text selectable by default (dezza) WEBUI: Add versioning to local preferences (Chocobo1) WEBUI: Make multi-rename search & replace fields use a monospace font (Atk) WEBUI: Fix wrong replacement sequence in IPv6 string (Chocobo1) WEBUI: Fix memory leak (bolshoytoster) WEBUI: Fix path autofill in set location and new category (tehcneko) RSS: Mark matched article as "read" if it refers to a duplicate torrent (glassez) WINDOWS: Update command line help message (KanishkaHalder1771) WINDOWS: NSIS: Don't require agreement on the license page (Chocobo1) LINUX: Fix preview not opening on Wayland (Isak05) LINUX: Add fallback for random number generator (Chocobo1) Download: qBittorrent 5.1.1 | Portable | ~40.0 MB (Open Source) Download: qBittorrent 64-bit installer (qt6) | 41.7 MB Links: qBittorrent Home page | Screenshot Get alerted to all of our Software updates on Twitter at @NeowinSoftware
  • Recent Achievements

    • Week One Done
      urbanmopdubai1 earned a badge
      Week One Done
    • One Month Later
      Jim Dugan earned a badge
      One Month Later
    • First Post
      Johnny Mrkvička earned a badge
      First Post
    • Week One Done
      viraltui earned a badge
      Week One Done
    • One Month Later
      serfegyed earned a badge
      One Month Later
  • Popular Contributors

    1. 1
      +primortal
      653
    2. 2
      Michael Scrip
      227
    3. 3
      ATLien_0
      218
    4. 4
      Steven P.
      150
    5. 5
      Xenon
      145
  • Tell a friend

    Love Neowin? Tell a friend!