A little bit of file de-duplication…

[goretsky Supervisor]

Hello,

I have been keeping an archive of all of the device drivers for all of the hardware I use over the years for a number of reasons:

• It is an easy and convenient way to roll back to an earlier version of something if the latest driver isn't working, especially if you don't have network access or limited network access.  
• I keep an archive of all of the device drivers for all of the hardware I use for a number of reasons:
• It is an easy and convenient way to roll back to an earlier version of something if the latest driver isn't working, especially if you don't have network access or limited network access.
• It lets you install drivers for older hardware that the manufacturer may not offer for download any more, without having to rely on third-party websites (some of dubious quality). That's even more important when the manufacturer is no longer in business.
• It can even be handy for security research, like trying to find out when a vulnerability was introduced (or fixed), validating a BYOVD attack, or as a clean set for fixing false positive detections.

This includes Nvidia's device drivers, which a while ago passed the size of an entire CD-ROM disc (650 MB), and currently sit at around 820-830 MB as downloaded from them.  And, I should note, that Nvidia releases separate installers for its desktop and notebook graphics chipsets, with filenames containing either desktop or notebook to denote what they were for.  Some drivers. like those released as a security patch or a hotfixe, contained the string desktop-notebook in the filename, indicating the package was for both types of hardware.

Despite having separate downloads for desktops and notebooks, the installers for the desktop and notebook versions of Nvidia's drivers have been the same size for a while.  This is something I did not pay too much attention to this, because I know from working at hardware and software companies that such things can be attributed to very small changes between product SKUs, like strings for product name or operating system names, detection logic for what to install and where to install it, and so forth.

Having just gone through process of buying yet another HDD to upgrade my backup system's capacity, I decided to take a look at those Nvidia driver downloads, since over time they have been taking up more and more space.  As much as I like having a library of drivers, I don't need duplicates in it.  And, since a large number of Nvidia's desktop and notebook device driver installation files seemed to be the same size, I figured it could be they were identical.  So, I decided to put this to the test and see if any of them were, in fact, duplicate files.

Using a tool to check for duplicate files, I was unsurprised to find that the installers for desktop and notebook versions of the installation packages generated identical CRC-32's and 32-bit checksums, indicating that the contents of the files were the same (I skipped MD5's, SHA-1's or stronger hashing methods as confirmation from two fast algorithms was sufficient).

Knowing that, it seems I could save myself quite a bit of disk space by deleting the duplicate files.  Between the Windows 10 and Windows 11 drivers, there were 288 GB of duplicate files.

I haven't looked to see if the driver installation packages perform any logic checks against their filename to determine whether to display that they are for desktop or graphics chipsets and go through the install process accordingly, but those are easy enough to check should I need to reinstall a particular package in the future.

Oh, the screen shot is from Funduc Software's Duplicate File Finder, just in case you're wondering. There are plenty of similar utilities you can find in Neowin's Software section (or you can script something yourself), but I have used a few of their utilities in the past so it was already installed on this PC when I decided to look at this.

Regards,

Aryeh Goretsky
 

 

Edited July 13, 2025 by goretsky
fixed a typo

[goretsky Supervisor]

Hello,

As a bit of a follow-up, I also decided to perform a file de-duplication against my software library, which I have been keeping for similar reasons to the hardware device driver library in the previous post.

While the results were not so dramatic, I did manage to trim some ~80GB of duplicate files, resulting in a small savings in storage as well as back-up time.

Regards,

Aryeh Goretsky
 

[ShadeOfBlue]

For large files like that that will never ever be modified, I like to just covert them to hardlinks with Czkawka. This preserves the organization and naming, while saving the disk space.

[goretsky Supervisor]

On 30/12/2025 at 20:39, ShadeOfBlue said:

For large files like that that will never ever be modified, I like to just covert them to hardlinks with Czkawka. This preserves the organization and naming, while saving the disk space.

Hello,

I decided that since the files themself were identical that I would instead use Nvidia's own naming scheme for HOTFIX drivers on the WHQL drivers, putting  desktop-notebook in the file's name in place of having two separate files with each word in the filename.

So, for example, instead of having

591.59-desktop-win10-win11-64bit-international-dch-whql.exe

and

591.59-notebook-win10-win11-64bit-international-dch-whql.exe

for the filename and its symlink, I now have

591.59-desktop-notebook-win10-win11-64bit-international-dch-whql.exe

for the filename.

This matches for structure used for the HOTFIX driver, to wit:

591.67-desktop-notebook-win10-win11-64bit-international-dch.hf.exe

So, if there is any filename-checking logic in the driver installer, this should handle it. 

The assumption I'm making here is that Nvidia does not have a completely separate process for making HOTFIX driver installation packages and that they instead use the same tooling and infrastructure as their WHQL drivers. 

So far, everything has worked out well for me, but I'll definitely report if things change.

Regards,

Aryeh Goretsky