• 0

Software to detect duplicates, with a difference?


Question

So to cut out a load of boring waffle for you, I need software to detect duplicate files.

 

I've tried Auslogics (as it was free) and it seemed to work well - for 100% identical files.

 

However, I have a ton of files that are 99% identical. By that I mean I have files that are basically the same but they have a -1 at the end of the file name. The rest of the file name is the same, the file size is the same, if they're a video then the duration is the same - but one will have a -1.

 

So we're talking about a file like: mycat.jpg and then another file called mycat-1.jpg ... where they're the exact same picture.

 

Is there such a software for Windows that can detect duplicates in this manner? I swear I must have many GB I could free up. I've already done 10GB off of 1 small subfolder.

8 answers to this question

Recommended Posts

  • 0

I used HashMyFiles to scan MD5 hashes (other hash types selectable) in various directories to scan for duplicate photos. Worked great. Name of file was irrelevant. You can open/scan multiple directories simultaneously. Duplicate hashes are handily color coded. Just picked which one to delete and done.

 

https://www.nirsoft.net/utils/hash_my_files.html

 

 

HashMyFiles.png

Edited by JustGeorge
  • Like 3
  • 0

If this is specific to digital camera photo's, check out https://www.advancedrenamer.com/

 

(Free for personal use)

 

It plugins to exif command line, and you can rename files to be Camera Model, Shutter Count (provided the camera embeds the the info).

 

I do like JustGeorge suggestion and taking a look at that myself for my various back up archive files.

  • 0

Thanks for the suggestions guys.

 

Turns out the software I was using, Auslogics, which is also free, does exactly what I need.

 

Issue was - I just didn't know how to use it properly at the point of asking.

 

Got it sorted now. Scanned 2 folders already & 10GB of duplicates in each for a 20GB free up.

  • 0

I used AllDup portable for video and images, but it isn't great, needs multiple passes to be able to pick out all dupes.

 

I will have to give a few of the suggestions in this topic a go.

  • 0
  On 23/09/2022 at 16:37, Technique said:

So to cut out a load of boring waffle for you, I need software to detect duplicate files.

 

I've tried Auslogics (as it was free) and it seemed to work well - for 100% identical files.

 

However, I have a ton of files that are 99% identical. By that I mean I have files that are basically the same but they have a -1 at the end of the file name. The rest of the file name is the same, the file size is the same, if they're a video then the duration is the same - but one will have a -1.

 

So we're talking about a file like: mycat.jpg and then another file called mycat-1.jpg ... where they're the exact same picture.

 

Is there such a software for Windows that can detect duplicates in this manner? I swear I must have many GB I could free up. I've already done 10GB off of 1 small subfolder.

Expand  

If the files are the same, the filename shouldn't matter because filenames don't affect checksums, so you would think Auslogics would detect it as being a duplicate anyway.

 

Just saw your follow-up comment, glad you got it sorted, 🙂

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Recently Browsing   0 members

    • No registered users viewing this page.
  • Posts

    • First thing I wondered as well. I miss the old color. I always thought it was nice how each app had its own color. Then they go and take 2 of the most commonly used apps and give them the same color.
    • We humans have to pass tests and get licenses to be considered safe to drive a car, but Tesla can just go "Trust me bro". Gotta' love it. ]
    • Last Chance to download The Inclusion Equation: Leveraging Data & AI (worth $21) for free by Steven Parker Claim your complimentary eBook worth $21 for free, before the offer ends today on June 24. The Inclusion Equation is a comprehensive, one-of-a-kind guide to merging DEI and employee wellbeing concepts with data analytics and AI. In this book, renowned thought leader and professional keynote speaker Dr. Serena Huang explains exactly how to quantify the effectiveness of new talent strategies by connecting them to a firm ROI estimate, enabling readers to approach and win the favor of higher-ups in any organization with the same effectiveness that marketing and financial departments do. This book is written in a style that is appealing and accessible to all readers regardless of technical background, but with enough depth to provide real insight and strategies. Dr. Serena H. Huang distills her 10 years of Fortune 500 people analytics leadership experience into tools and framework you can leverage to measure and improve DEI and wellbeing in your workplace. Some of the topics explored in this book include: Attract and retain top talent, including Gen Z and Millennials, with tailored DEI and wellbeing strategies Quantifying not only a talent strategy's perceived initial effect on an organization, but also its improvement and expansion over time Turning DEI and wellbeing from illusive corporate concepts to quantifiable metrics Harness the power of AI to create synchronized DEI and wellbeing strategies that maximize ROI Getting serious attention from your CEO and CFO by quantifying HR initiatives Using data storytelling to demonstrate the business impact of DEI and wellbeing Preparing for the future by understanding the role of AI in creating an inclusive and healthy workplace The Inclusion Equation is a complete guide for DEI and wellbeing, covering getting started in measurement to using storytelling to influence leadership. This is the contemporary playbook for any organization intending to substantially improve their diversity, equity, inclusion, and employee wellbeing by leveraging data & AI. This book is also perfect for any data analytics professionals who want to understand how to apply analytics to issues that keep their CEOs up at night. Whether you are a data expert or data novice, as long as you are serious about improving DEI and wellbeing, this book is for you. This free to download offer expires June 24. How to get it Please ensure you read the terms and conditions to claim this offer. Complete and verifiable information is required in order to receive this free offer. If you have previously made use of these free offers, you will not need to re-register. While supplies last! Download The Inclusion Equation: Leveraging Data & AI (worth $21) for free Offered by Wiley, view other free resources The below offers are also available for free in exchange for your (work) email: Last Chance: Excel Quick and Easy ($12 Value) FREE – Expires 6/24 Last Chance: The Inclusion Equation: Leveraging Data & AI ($21 Value) FREE – Expires 6/24 Microsoft 365 Copilot At Work ($60 Value) FREE – Expires 6/25 Natural Language Processing with Python ($39.99 Value) FREE – Expires 6/25 How to Engage Buyers and Drive Growth in the Age of AI ($22.95 Value) FREE – Expires 7/1 Using Artificial Intelligence to Save the World ($30.00 Value) FREE – Expires 7/1 Essential: How Distributed Teams, Generative AI, [...] ($18.00 Value) FREE – Expires 7/2 The Chief AI Officer's Handbook: Master AI leadership with strategies to innovate, overcome challenges, and drive business growth ($9.99 Value) FREE for a Limited Time – Expires 7/2 The Ultimate Linux Newbie Guide – Featured Free content Python Notes for Professionals – Featured Free content Learn Linux in 5 Days – Featured Free content Quick Reference Guide for Cybersecurity – Featured Free content We post these because we earn commission on each lead so as not to rely solely on advertising, which many of our readers block. It all helps toward paying staff reporters, servers and hosting costs. Other ways to support Neowin The above deal not doing it for you, but still want to help? Check out the links below. Check out our partner software in the Neowin Store Buy a T-shirt at Neowin's Threadsquad Subscribe to Neowin - for $14 a year, or $28 a year for an ad-free experience Disclosure: An account at Neowin Deals is required to participate in any deals powered by our affiliate, StackCommerce. For a full description of StackCommerce's privacy guidelines, go here. Neowin benefits from shared revenue of each sale made through the branded deals site.
    • I knew it had to be fan art because it actually looked pretty cool.
  • Recent Achievements

    • Dedicated
      Parallax Abstraction earned a badge
      Dedicated
    • First Post
      956400 earned a badge
      First Post
    • Week One Done
      davidfegan earned a badge
      Week One Done
    • First Post
      Ainajohn earned a badge
      First Post
    • Conversation Starter
      sophiaisabella32 earned a badge
      Conversation Starter
  • Popular Contributors

    1. 1
      +primortal
      593
    2. 2
      ATLien_0
      223
    3. 3
      Michael Scrip
      170
    4. 4
      +FloatingFatMan
      152
    5. 5
      Som
      135
  • Tell a friend

    Love Neowin? Tell a friend!