Jump to content



Photo

Duplicate File Finder

codelake duplicate file finder

  • Please log in to reply
29 replies to this topic

#16 OP wrack

wrack

    Wireless Robotic Android Calibrated for Killing

  • Joined: 09-December 06
  • Location: Melbourne, Australia

Posted 03 July 2014 - 03:00

New version (1.2014.0703.0246) is up.

 

Get it http://www.codelake....eFileFinder.zip

 

* Add option to open file location in the right click menu
* Add option to choose the combination of properties when performing simple search
* Add option of intelligent hash matching when performing advanced search
* Add option to specify normal file search filter
* Add option to specify regular expression file search filter
* Add option to just find files based on the filter options (Not looking for duplicates)
* Add smart help button and other cosmetic changes




#17 OP wrack

wrack

    Wireless Robotic Android Calibrated for Killing

  • Joined: 09-December 06
  • Location: Melbourne, Australia

Posted 03 July 2014 - 04:47

New version (1.2014.0703.0436) is up.

 

Get it http://www.codelake....eFileFinder.zip

 

* Improved performance using multithreading.



#18 StarkWiz

StarkWiz

    Neowinian

  • Joined: 28-November 06
  • Location: Mumbai, India
  • OS: Windows 8 Pro 64-bit

Posted 03 July 2014 - 06:59

I have just started using your utility and I liked the interface, will let you know my more suggestions.

 

You can save a lot of time required for initial scan by using MFT as it already has most of the data you need, just like an index.

It maybe difficult to implement the support for that though, but it's worth the effort considering the no. of files we have these days.

 

Just check this WizTree software, it can also save MFT to a file.
https://antibody-sof...our-hard-drive/

I was able to get disk usage of 4 TB disk with approx. 1 million files in 1 min. Normally it would take lot of time.

There are few other software which make use of MFT as well.

 

While writing this post, it threw up an exception during scan as below and exited.

----- < CodeLake Exception Message > --------------------------------------------
An unexpected error has occurred. Application.ThreadException


----- < CodeLake Exception Details > --------------------------------------------
System.NullReferenceException: Object reference not set to an instance of an object.
   at System.Windows.Forms.Control.MarshaledInvoke(Control caller, Delegate method, Object[] args, Boolean synchronous)
   at System.Windows.Forms.Control.Invoke(Delegate method, Object[] args)
   at CodeLake.Utilities.DuplicateFileFinder.FormMain.a(Object A_0, RunWorkerCompletedEventArgs A_1)
   at System.ComponentModel.BackgroundWorker.OnRunWorkerCompleted(RunWorkerCompletedEventArgs e)
   at System.ComponentModel.BackgroundWorker.AsyncOperationCompleted(Object arg)

I ran it few times after posting this message initially.

I noticed that it throws exception exactly after finishing the initial scan or just before it is going to find the duplicate files.

I tried running it as admin and normal it failed the same way.


Edited by StarkWiz, 03 July 2014 - 07:06.


#19 Nashy

Nashy

    Neowinian Senior

  • Joined: 05-September 04
  • Location: Brisbane, Australia
  • OS: Windows 8.1
  • Phone: Samsung Galaxy S5 - SM-G900i

Posted 03 July 2014 - 07:08

Will try this!  Thanks mate.



#20 OP wrack

wrack

    Wireless Robotic Android Calibrated for Killing

  • Joined: 09-December 06
  • Location: Melbourne, Australia

Posted 03 July 2014 - 22:10

StarkWiz, on 03 Jul 2014 - 16:59, said:

I have just started using your utility and I liked the interface, will let you know my more suggestions.

 

You can save a lot of time required for initial scan by using MFT as it already has most of the data you need, just like an index.

It maybe difficult to implement the support for that though, but it's worth the effort considering the no. of files we have these days.

 

Just check this WizTree software, it can also save MFT to a file.
https://antibody-sof...our-hard-drive/

I was able to get disk usage of 4 TB disk with approx. 1 million files in 1 min. Normally it would take lot of time.

There are few other software which make use of MFT as well.

 

While writing this post, it threw up an exception during scan as below and exited.

----- < CodeLake Exception Message > --------------------------------------------
An unexpected error has occurred. Application.ThreadException


----- < CodeLake Exception Details > --------------------------------------------
System.NullReferenceException: Object reference not set to an instance of an object.
   at System.Windows.Forms.Control.MarshaledInvoke(Control caller, Delegate method, Object[] args, Boolean synchronous)
   at System.Windows.Forms.Control.Invoke(Delegate method, Object[] args)
   at CodeLake.Utilities.DuplicateFileFinder.FormMain.a(Object A_0, RunWorkerCompletedEventArgs A_1)
   at System.ComponentModel.BackgroundWorker.OnRunWorkerCompleted(RunWorkerCompletedEventArgs e)
   at System.ComponentModel.BackgroundWorker.AsyncOperationCompleted(Object arg)

I ran it few times after posting this message initially.

I noticed that it throws exception exactly after finishing the initial scan or just before it is going to find the duplicate files.

I tried running it as admin and normal it failed the same way.

 

Thanks for that exception report. Could you please do 1 more testing on that same set of folder(s) but this time just click on Find Files? If that works then the error is most probably due to multithreading I introduced. Will be hard to debug but I shall do my best.



#21 OP wrack

wrack

    Wireless Robotic Android Calibrated for Killing

  • Joined: 09-December 06
  • Location: Melbourne, Australia

Posted 03 July 2014 - 22:33

Nashy, on 03 Jul 2014 - 17:08, said:

Will try this!  Thanks mate.

 

Welcome :)



#22 xrobwx

xrobwx

    Leave the gun. Take the cannoli.

  • Tech Issues Solved: 1
  • Joined: 14-June 03
  • Location: Panama City Beach, FL USA
  • OS: Win 8.1
  • Phone: Galaxy Note II

Posted 03 July 2014 - 22:50

I got an error as well. Very nice piece of work!

----- < CodeLake Exception Message > --------------------------------------------
An unexpected error has occurred. Application.ThreadException


----- < CodeLake Exception Details > --------------------------------------------
System.NullReferenceException: Object reference not set to an instance of an object.
   at System.Windows.Forms.Control.MarshaledInvoke(Control caller, Delegate method, Object[] args, Boolean synchronous)
   at System.Windows.Forms.Control.Invoke(Delegate method, Object[] args)
   at CodeLake.Utilities.DuplicateFileFinder.FormMain.a(Object A_0, RunWorkerCompletedEventArgs A_1)
   at System.ComponentModel.BackgroundWorker.OnRunWorkerCompleted(RunWorkerCompletedEventArgs e)
   at System.ComponentModel.BackgroundWorker.AsyncOperationCompleted(Object arg)

My system specs:

Computer:      GIGABYTE X58A-UD3R
CPU:           Intel Core i7-980 (Gulftown, B1)
               3333 MHz (25.00x133.3) @ 3899 MHz (27.00x144.4)
Motherboard:   GIGABYTE X58A-UD3R
Chipset:       Intel X58 (Tylersburg 36S) + ICH10R
Memory:        24576 MBytes @ 722 MHz, 10.0-9-9-24
               - 4096 MB PC10600 DDR3 SDRAM - A-DATA Technology 
               - 4096 MB PC10600 DDR3 SDRAM - A-DATA Technology 
               - 4096 MB PC10600 DDR3 SDRAM - A-DATA Technology 
               - 4096 MB PC10600 DDR3 SDRAM - A-DATA Technology 
               - 4096 MB PC10600 DDR3 SDRAM - A-DATA Technology 
               - 4096 MB PC10600 DDR3 SDRAM - A-DATA Technology 
Graphics:      EVGA e-GeForce GTX 460 SE
               NVIDIA GeForce GTX 460 SE, 1024 MB GDDR5 SDRAM
Graphics:      EVGA e-GeForce GTX 460 SE
               NVIDIA GeForce GTX 460 SE, 1024 MB GDDR5 SDRAM
Drive:         Hitachi HDT721010SLA360, 976.8 GB, Serial ATA 3Gb/s
Drive:         ST2000DL003-9VT166, 1953.5 GB, Serial ATA 6Gb/s @ 3Gb/s
Drive:         WDC WD1002FAEX-00Z3A0, 976.8 GB, Serial ATA 6Gb/s
Drive:         HP DVD Writer 300n, DVD+R Writer
Sound:         Intel ICH10 - High Definition Audio Controller [A0]
Sound:         NVIDIA GF104 - High Definition Audio Controller
Sound:         NVIDIA GF104 - High Definition Audio Controller
Network:       RealTek Semiconductor RTL8168/8111 PCI-E Gigabit Ethernet NIC
OS:            Microsoft Windows 8.1 Professional (x64) Build 9600
 


#23 StarkWiz

StarkWiz

    Neowinian

  • Joined: 28-November 06
  • Location: Mumbai, India
  • OS: Windows 8 Pro 64-bit

Posted 03 July 2014 - 23:58

Thanks for that exception report. Could you please do 1 more testing on that same set of folder(s) but this time just click on Find Files? If that works then the error is most probably due to multithreading I introduced. Will be hard to debug but I shall do my best.

Got exactly same error.

I tried on different drives and noticed that I dont get this error on drives with less no. of files.

The drive on which I got error has around 82k files.



#24 OP wrack

wrack

    Wireless Robotic Android Calibrated for Killing

  • Joined: 09-December 06
  • Location: Melbourne, Australia

Posted 04 July 2014 - 00:02

Thanks for the bug report guys. I have found it to be due to the multithreading for sure but I can't reproduce it and so very hard to debug. For the time being, I am reverting that code change.

 

New version (1.2014.0703.2350) is up.

 

Get it http://www.codelake....eFileFinder.zip

 

* Removed multithreading due to reliability issues.
* Change the min and max size options and made them independent of one another.



#25 OP wrack

wrack

    Wireless Robotic Android Calibrated for Killing

  • Joined: 09-December 06
  • Location: Melbourne, Australia

Posted 04 July 2014 - 00:06

Got exactly same error.

I tried on different drives and noticed that I dont get this error on drives with less no. of files.

The drive on which I got error has around 82k files.

Thanks for the stats.

 

I ran advanced search on a 36.9GB single folder with 436639 files and 61814 subfolder and it took 1 hour to find me all duplicate files but no errors so far. I have reverted the multithreading code anyways so shouldn't be an issue for now.

 

Apologies for any troubles caused by this.



#26 StarkWiz

StarkWiz

    Neowinian

  • Joined: 28-November 06
  • Location: Mumbai, India
  • OS: Windows 8 Pro 64-bit

Posted 04 July 2014 - 01:25



Thanks for the stats.

I ran advanced search on a 36.9GB single folder with 436639 files and 61814 subfolder and it took 1 hour to find me all duplicate files but no errors so far. I have reverted the multithreading code anyways so shouldn't be an issue for now.

Apologies for any troubles caused by this.


I'll try the new version. Maybe we are missing some files required for multi threading?

#27 OP wrack

wrack

    Wireless Robotic Android Calibrated for Killing

  • Joined: 09-December 06
  • Location: Melbourne, Australia

Posted 04 July 2014 - 04:10

StarkWiz, on 04 Jul 2014 - 11:25, said:StarkWiz, on 04 Jul 2014 - 11:25, said:

I'll try the new version. Maybe we are missing some files required for multi threading?

 

Nope. The multithreading didn't give me any errors but also produced inconsistent results. Basically only thing lost from the lack of multithreading is the performance boost.



#28 astropheed

astropheed

    astropheed

  • Tech Issues Solved: 2
  • Joined: 08-December 11
  • Location: Sydney, AU

Posted 04 July 2014 - 04:26

Nope. The multithreading didn't give me any errors but also produced inconsistent results. Basically only thing lost from the lack of multithreading is the performance boost.

 

I doubt their would be much benefit, the CPU isn't the bottleneck by far.



#29 OP wrack

wrack

    Wireless Robotic Android Calibrated for Killing

  • Joined: 09-December 06
  • Location: Melbourne, Australia

Posted 04 July 2014 - 04:56

Garnet H., on 04 Jul 2014 - 14:26, said:

I doubt their would be much benefit, the CPU isn't the bottleneck by far.

 

My initial test showed the scanning performance improvement by 200-400% specifically in Advanced Search as it is calculating MD5 hash. I would very much prefer to use multithreading but for the time being, I can live without it.



#30 OP wrack

wrack

    Wireless Robotic Android Calibrated for Killing

  • Joined: 09-December 06
  • Location: Melbourne, Australia

Posted 21 July 2014 - 02:05

New version (1.2014.0721.0103) is up.

 

Get it http://www.codelake....eFileFinder.zip

 

* Fix issues with min and max size options
* Fix issues with Intelligent Hash Matching scanning

* Add ability to select from multiple drives (Currently limited to 3 different drives)

* Ui and general code cleanup