Why different free spaces ? (Same hdd size & files).

[bigmehdi]

Hi,
I have a lossless music collection on a 1 Tb hard drive. I made a copy of everything to an external hard drive of 1 Tb too. But the free available space is not the same (1 gb of difference !) .I  know that with my current pc formatting the external hard drive, and re-copying everything again would solve the issue. I've done this once.  But sometimes I like to update my collection on the external drive  (from an other pc) ,  and when I resynch again to replicate the changes on my local hard drive, a difference of 1 gb for the free space comes back.   I  want to understand where the difference of 1 Gb comes from. I  tried to delete/disable any potential "restore point" (as this could impact the "System Volume Information" folder)  ,  but this makes no difference.  Also I  checked for any hidden files, found nothing. Virus scanning, nothing.

If this could help I've got some info using ms-dos below. G: is the local drive, J: is an external one.

 

C:\Windows\system32>fsutil fsinfo ntfsinfo  j:
NTFS Volume Serial Number :       0x8c845d4b845d38c2
Version :                         3.1
Number Sectors :                  0x00000000746f5fff
Total Clusters :                  0x000000000e8debff
Free Clusters  :                  0x000000000169588b
Total Reserved :                  0x0000000000049640
Bytes Per Sector  :               512
Bytes Per Physical Sector :       <Not Supported>
Bytes Per Cluster :               4096
Bytes Per FileRecord Segment    : 1024
Clusters Per FileRecord Segment : 0
Mft Valid Data Length :           0x0000000003400000
Mft Start Lcn  :                  0x00000000000c0000
Mft2 Start Lcn :                  0x0000000000000002
Mft Zone Start :                  0x00000000000c3400
Mft Zone End   :                  0x00000000000cc820
RM Identifier:        D50C9E71-D468-11E3-8F3F-001D7D06BBDC

C:\Windows\system32>fsutil fsinfo ntfsinfo  g:
NTFS Volume Serial Number :       0x3eee02c4ee027485
Version :                         3.1
Number Sectors :                  0x00000000747057ff
Total Clusters :                  0x000000000e8e0aff
Free Clusters  :                  0x0000000001695b52
Total Reserved :                  0x00000000000003f0
Bytes Per Sector  :               512
Bytes Per Physical Sector :       512
Bytes Per Cluster :               4096
Bytes Per FileRecord Segment    : 1024
Clusters Per FileRecord Segment : 0
Mft Valid Data Length :           0x0000000003c00000
Mft Start Lcn  :                  0x00000000000c0000
Mft2 Start Lcn :                  0x0000000000000002
Mft Zone Start :                  0x00000000000c3c00
Mft Zone End   :                  0x00000000000cc820
RM Identifier:        8D498DA5-4CED-11E2-BDC3-001D7D06BBDC

Any suggestion/explanation ?

Thanks
 

[+BudMan MVC]

I would assume its the Reserved Space that your seeing the difference in

By default NTFS reserves zone, 12.5% of volume size for MFT and does not allow writing there any user's data, which lets MFT to grow. However, when, for example, a lot of files are placed to the drive, MFT can grow beyond the reserved zone and becomes fragmented.

If your drive becomes full that can shrink below that number original reserved number if not being used - like lots of BIG files. Did your original disk become full at some point?

If doing the math right

Total Reserved : 0x0000000000049640 = 300608 x 4096 = 1,231,290,368 Bytes

The other disk is only 1008 x 4096 = 4,128,768

If you fill up the space, then the MFT should shrink.. Couple different ways to do that. But not sure why you would be worried about it?? Its 1GB out of 1TB ;)

edit: If you run defrag in cmd prompt you can get a report

defrag d: -A -V

Master File Table (MFT):

MFT size = 117.00 MB

MFT record count = 119807

MFT usage = 100%

Total MFT fragments = 2

So what does it show for your MFT for your 2 drives?

[bigmehdi]

hi,
 

The other disk is only 1008 x 4096 = 4,128,768

Well, I  don't understand well this number.

 

But not sure why you would be worried about it?? Its 1GB out of 1TB ;)

 

Yes 1 Gb from 1Tb is not big, but at least I'm reassured than nothing wrong is happening.

According to your comments, I  guess  the difference is because of fragmentation ?
There was a time, 1 Gb meant a lot for me (especially with a hard drive of 8 gb :) ).
 

: If you run defrag in cmd prompt you can get a report

 

Well, the reported free space is not the same on explorer and with the dos command.

On explorer external hard drive J: indicate 89.1 gb of free space, while with dos command I get 90.33 gb

Here is the full dos report:

C:\Windows\system32>defrag g: -A -V
Microsoft Disk Defragmenter
Copyright (c) 2007 Microsoft Corp.

Invoking analysis on Musik (G:)...


The operation completed successfully.

Post Defragmentation Report:

        Volume Information:
                Volume size                 = 931,50 GB
                Cluster size                = 4 KB
                Used space                  = 841,17 GB
                Free space                  = 90,33 GB

        Fragmentation:
                Total fragmented space      = 1%
                Average fragments per file  = 2,32

                Movable files and folders   = 53483
                Unmovable files and folders = 10

        Files:
                Fragmented files            = 523
                Total file fragments        = 62732

        Folders:
                Total folders               = 4622
                Fragmented folders          = 409
                Total folder fragments      = 702

        Free space:
                Free space count            = 1748
                Average free space size     = 52,83 MB
                Largest free space size     = 79,61 GB

        Master File Table (MFT):
                MFT size                    = 60,00 MB
                MFT record count            = 61439
                MFT usage                   = 100%
                Total MFT fragments         = 1

        Note: File fragments larger than 64MB are not included in the fragmentation statistics.

        You do not need to defragment this volume.

C:\Windows\system32>defrag j: -A -V
Microsoft Disk Defragmenter
Copyright (c) 2007 Microsoft Corp.

Invoking analysis on Musik (J:)...


The operation completed successfully.

Post Defragmentation Report:

        Volume Information:
                Volume size                 = 931,47 GB
                Cluster size                = 4 KB
                Used space                  = 841,14 GB
                Free space                  = 90,33 GB

        Fragmentation:
                Total fragmented space      = 0%
                Average fragments per file  = 1,00

                Movable files and folders   = 52516
                Unmovable files and folders = 10

        Files:
                Fragmented files            = 14
                Total file fragments        = 25

        Folders:
                Total folders               = 4619
                Fragmented folders          = 859
                Total folder fragments      = 1255

        Free space:
                Free space count            = 59
                Average free space size     = 1,52 GB
                Largest free space size     = 69,66 GB

        Master File Table (MFT):
                MFT size                    = 52,00 MB
                MFT record count            = 53247
                MFT usage                   = 100%
                Total MFT fragments         = 3

        Note: File fragments larger than 64MB are not included in the fragmentation statistics.

        You do not need to defragment this volume.

[+BudMan MVC]

Free space = 90,33 GB

That looks the same on both..

You also have more files on one than the other

Movable files and folders = 52516

Movable files and folders = 53483

"The other disk is only 1008 x 4096 = 4,128,768"

Convert the hex number..

Total Reserved : 0x00000000000003f0

So 3F0 = (3 * 16^2) + (15 (F) * 16^1) + (0 * 16^0)

3 x 16^2 = 3 x 256 = 768

15 x 16^1 = 15 * 16 = 240

0 x 16^1 = 0 x 1 = 0

Add them up, then x 4096 is the Bytes per cluster

Its not fragmentation - it is the reserved space for the MFT.. If you look at your MFT size they are different - which makes sense with different files. But that is not really showing you the reserved space.. Let me see if I can dig up a easy way to show that. Other than doing the math on your fsutil output.

[bigmehdi]

You also have more files on one than the other

Movable files and folders = 52516

Movable files and folders = 53483

But when I select all files/folder at the root, and then look at properties I get same number of file and folders from explorer i.e: 47 436 files, 4 605 folders for both hard drive . Only the hidden system folder are not selected ( System Volume Information & recycle bin).

Also I synched both hard drive using a comparison tool "beyond compare", and off course the comparison indicate no differences in files/ folders.

 

.. If you look at your MFT size they are different - which makes sense with different files

But again files are the same, and using a comparison tool I can't find a difference.

[+BudMan MVC]

well why does the defrag report show differences in number of files?

Movable files and folders = 52516

Movable files and folders = 53483

And its just not a couple - its almost 1k

While I can understand the curiosity - I do!! Your talking 1GB out of a TB disk.. that is like .1%

You have a difference in the reserved space clearly from your fsutil between disks. But it seems high to me - how did you format this disk? What OS are you using?

Can you output

C:\>fsutil behavior query mftzone

MftZone = 0

So if I look at this system at work

C:\>fsutil fsinfo ntfsinfo c:

NTFS Volume Serial Number : 0x411dc95f01904c54

Version : 3.1

Number Sectors : 0x00000000253fda70

Total Clusters : 0x0000000004a7fb4e

Free Clusters : 0x00000000032c27f9

Total Reserved : 0x0000000000000780

Bytes Per Sector : 512

Bytes Per Cluster : 4096

Bytes Per FileRecord Segment : 1024

Clusters Per FileRecord Segment : 0

Mft Valid Data Length : 0x000000000d5c0000

Mft Start Lcn : 0x0000000000000004

Mft2 Start Lcn : 0x0000000002448cd8

Mft Zone Start : 0x0000000001067e40

Mft Zone End : 0x00000000010683a0

RM Identifier: 61A8CFCA-D4F7-11E2-9177-806E6F6E6963

----

Total Reserved : 0x0000000000000780

30720 x 4096 = 125,829,120

If I recall in windows 7 mftzone of 0 = 200MB.. So I might not recalling what that reserved space is reporting on?? But odd how your saying your seeing 1GB difference and that sure right in the ball park..

Can you run this command for your disks

C:\>fsutil volume diskfree c:

Total # of free bytes : 217998594048

Total # of bytes : 319970140160

Total # of avail free bytes : 217998594048

That reserved space is space for the ntfs meta data, this can contain lots of different stuff

I would be most interested in your log file size..

run a chkdsk on the disks and you should get the log file size

Windows has checked the file system and found no problems.

312470840 KB total disk space.

99675336 KB in 175463 files.

87688 KB in 27449 indexes.

0 KB in bad sectors.

331768 KB in use by the system.

65536 KB occupied by the log file.

212376048 KB available on disk.

4096 bytes in each allocation unit.

78117710 total allocation units on disk.

53094012 allocation units available on disk.

What is the reports for your 2 disks from chkdsk?

[bigmehdi]

 how did you format this disk? What OS are you using?

 

Actually I use two Pc, I tried to simplify thing by avoiding unecessary details.

Let's call one pc A, and the other B. I move sometimes from a town to an other , and make change to the musical collection.

So the portable hard drive, is the way for me to resynch again files/folder changes, when moving between locations.

The problem occured in the pc B, I  made a little change in local drive, and then free available space was abruptly decreased of 1 gb.

Afer resynching with the external hard drive, it was obvious that 1 gb was missing.

I  think I  reformatted my external drive there, and then after copying everything I  got this time the same free space size (i.e 1 gb lacking for the free space).

It's once I  got back to pc A, and then resynched using the tool "beyond compare", that I  got to admit that 1 gb mysteriously disappeared from my external drive.

 

I thought it could be a very hard to detect rootkit, that's why I wanted to inspect this.

 

Both pc are under win 7 , 64 bits. Formatting done by right clicking, and default options.

Currently I use the pc I called A.

 

Can you output

C:\>fsutil behavior query mftzone

MftZone = 0

 

 

I get same output, when I type this i.e MftZone = 0.

 

 

Can you run this command for your disks

C:\>fsutil volume diskfree c:

 

 

 

well I get:

 

C:\Windows\system32>fsutil volume diskfree  g:

Total # of free bytes        : 96992632832

Total # of bytes             : 1000202039296

Total # of avail free bytes  : 96992632832

C:\Windows\system32>fsutil volume diskfree  j:

Total # of free bytes        : 95762558976

Total # of bytes             : 1000169533440

Total # of avail free bytes  : 95762558976

 

 

There's a small size difference between hard drives if I compare "total of bytes", but it's around 31 Mb  ( far from 1 Gb).

 

What is the reports for your 2 disks from chkdsk?

 

 

 

Local drive G:

C:\Windows\system32>chkdsk g:
The type of the file system is NTFS.
Volume label is Musik.

WARNING!  F parameter not specified.
Running CHKDSK in read-only mode.

CHKDSK is verifying files (stage 1 of 3)...
  61440 file records processed.
File verification completed.
  6813 large file records processed.
  0 bad file records processed.
  0 EA records processed.
  0 reparse records processed.
CHKDSK is verifying indexes (stage 2 of 3)...
  70680 index entries processed.
Index verification completed.
  0 unindexed files scanned.
  0 unindexed files recovered.
CHKDSK is verifying security descriptors (stage 3 of 3)...
  61440 file SDs/SIDs processed.
Security descriptor verification completed.
  4621 data files processed.
CHKDSK is verifying Usn Journal...
  37710080 USN bytes processed.
Usn Journal verification completed.
Windows has checked the file system and found no problems.

 976759807 KB total disk space.
 884887796 KB in 47448 files.
     21928 KB in 4622 indexes.
         0 KB in bad sectors.
    194591 KB in use by the system.
     65536 KB occupied by the log file.
  91655492 KB available on disk.

      4096 bytes in each allocation unit.
 244189951 total allocation units on disk.
  22913873 allocation units available on disk.

External drive J:

C:\Windows\system32>chkdsk j:
The type of the file system is NTFS.
Volume label is Musik.

WARNING!  F parameter not specified.
Running CHKDSK in read-only mode.

CHKDSK is verifying files (stage 1 of 3)...
  53248 file records processed.
File verification completed.
  0 large file records processed.
  0 bad file records processed.
  0 EA records processed.
  0 reparse records processed.
CHKDSK is verifying indexes (stage 2 of 3)...
  62482 index entries processed.
Index verification completed.
  0 unindexed files scanned.
  0 unindexed files recovered.
CHKDSK is verifying security descriptors (stage 3 of 3)...
  53248 file SDs/SIDs processed.
Security descriptor verification completed.
  4618 data files processed.
CHKDSK is verifying Usn Journal...
  17853760 USN bytes processed.
Usn Journal verification completed.
Windows has checked the file system and found no problems.

 976728063 KB total disk space.
 882274612 KB in 47444 files.
     20616 KB in 4619 indexes.
         0 KB in bad sectors.
    166935 KB in use by the system.
     65536 KB occupied by the log file.
  94265900 KB available on disk.

      4096 bytes in each allocation unit.
 244182015 total allocation units on disk.
  23566475 allocation units available on disk.

Hence same log size.

[+BudMan MVC]

clearly these have different number of files

CHKDSK is verifying files (stage 1 of 3)...

53248 file records processed.

CHKDSK is verifying files (stage 1 of 3)...

61440 file records processed.

884887796 KB in 47448 files.

882274612 KB in 47444 files.

194591 KB in use by the system.

166935 KB in use by the system.

These disks are showing all kinds of differences..

if you do the math here

884887796 KB in 47448 files.

882274612 KB in 47444 files.

You have 2,613,184 KB more stored on 1 disk than another..

[bigmehdi]

clearly these have different number of files

 

But when I use a comparison tool like "beyond  compare", it detects nothing.

Just tried the freeware "freefilesync" to perform a comparison, and no difference too.

 

Also I tried to list all hidden files, and nothing significant found.

 

External drive J:

C:\Windows\system32>dir /s j: /A:H
 Volume in drive J is Musik
 Volume Serial Number is 845D-38C2

 Directory of J:\

10/09/2014  23:40    <DIR>          $RECYCLE.BIN
30/09/2014  17:54    <DIR>          System Volume Information
               0 File(s)              0 bytes

 Directory of J:\$RECYCLE.BIN

10/09/2014  23:40    <DIR>          .
10/09/2014  23:40    <DIR>          ..
10/09/2014  23:40    <DIR>          S-1-5-21-2840249066-1808354765-502258188-100
1
               0 File(s)              0 bytes

 Directory of J:\$RECYCLE.BIN\S-1-5-21-2840249066-1808354765-502258188-1001

10/09/2014  23:40    <DIR>          .
10/09/2014  23:40    <DIR>          ..
10/09/2014  23:40               129 desktop.ini
               1 File(s)            129 bytes

     Total Files Listed:
               1 File(s)            129 bytes
               7 Dir(s)  95 762 558 976 bytes free

Local drive G:

C:\Windows\system32>dir /s g: /A:H
 Volume in drive G is Musik
 Volume Serial Number is EE02-7485

 Directory of G:\

11/01/2013  22:08    <DIR>          $RECYCLE.BIN
30/09/2014  17:29    <DIR>          System Volume Information
               0 File(s)              0 bytes

 Directory of G:\$RECYCLE.BIN

11/01/2013  22:08    <DIR>          .
11/01/2013  22:08    <DIR>          ..
10/01/2013  04:02    <DIR>          S-1-5-21-1361396589-2356774225-1261531009-10
01
04/01/2013  00:33    <DIR>          S-1-5-21-1361396589-2356774225-1261531009-10
03
10/01/2013  19:37    <DIR>          S-1-5-21-25329115-1288955257-3199750000-1001

29/09/2014  15:54    <DIR>          S-1-5-21-2840249066-1808354765-502258188-100
1
               0 File(s)              0 bytes

 Directory of G:\$RECYCLE.BIN\S-1-5-21-1361396589-2356774225-1261531009-1001

10/01/2013  04:02    <DIR>          .
10/01/2013  04:02    <DIR>          ..
24/12/2012  00:04               129 desktop.ini
               1 File(s)            129 bytes

 Directory of G:\$RECYCLE.BIN\S-1-5-21-1361396589-2356774225-1261531009-1003

04/01/2013  00:33    <DIR>          .
04/01/2013  00:33    <DIR>          ..
04/01/2013  00:33               129 desktop.ini
               1 File(s)            129 bytes

 Directory of G:\$RECYCLE.BIN\S-1-5-21-25329115-1288955257-3199750000-1001

10/01/2013  19:37    <DIR>          .
10/01/2013  19:37    <DIR>          ..
10/01/2013  19:37               129 desktop.ini
               1 File(s)            129 bytes

 Directory of G:\$RECYCLE.BIN\S-1-5-21-2840249066-1808354765-502258188-1001

29/09/2014  15:54    <DIR>          .
29/09/2014  15:54    <DIR>          ..
11/01/2013  22:08               129 desktop.ini
               1 File(s)            129 bytes

     Total Files Listed:
               4 File(s)            516 bytes
              16 Dir(s)  96 992 628 736 bytes free

[bigmehdi]

If I admit there's a difference, why the comparison tools can't find anything ? There"s something suspiscious.  I wish I could find it.

[+BudMan MVC]

well if you have stuff in the recycle bin.. Those could be quite large

example

10/01/2013 04:02 <DIR> S-1-5-21-1361396589-2356774225-1261531009-1001

04/01/2013 00:33 <DIR> S-1-5-21-1361396589-2356774225-1261531009-1003

10/01/2013 19:37 <DIR> S-1-5-21-25329115-1288955257-3199750000-1001

29/09/2014 15:54 <DIR> S-1-5-21-2840249066-1808354765-502258188-1001

So those 1001, 1003 are the RID of the user accounts that owns those recycled files.

You can go in via cmd line and clear those for sure - or look in them.

Your command is only showing hidden - but the files in the bin might not have H attribute set.

[bigmehdi]

well if you have stuff in the recycle bin.. Those could be quite large

 

Well, I can force a permanent delete of the hidden system Recycle folder, and this doesn't help me to recover the missing Gb. This folder, is automatically re-created , so I don't think it's a problem.

 

Sorry, can't see the pic you posted.

 

Otherwise do you think that a malware/rootkit could hide files from explorer ?

[+BudMan MVC]

1GB malware/rootkit - yeah I don't think so..

I just don't understand why your so worried about 1GB.. Clearly your disk is showing different sizes and different number of files.. If your worried about root kit on it, then wipe it and copy your files over again.

Yeah not seeing the image either - even though I saw it when I posted it.. They are having problems with images that is clear..

[bigmehdi]

Ok, you are right , this doesn't look like a big issue. Thanks for trying anyways.

[+BudMan MVC]

You notice the huge difference in fragmentation.. That could account for quite a bit of space actually..

Try defrag

Files:

Fragmented files = 523

Total file fragments = 62732

Files:

Fragmented files = 14

Total file fragments = 25

Keep in mind these drives are not the same out of the gate

Number Sectors : 0x00000000746f5fff = 1953456127

Total Clusters : 0x000000000e8debff

Number Sectors : 0x00000000747057ff = 1953519615

Total Clusters : 0x000000000e8e0aff

If you look at free clusters

23681163 - 23681874 = 711

There is so many things different with this disk, trying to figure out where 1 GB is crazy ;)