Robocopy File Transfer is it suppose to take 24 Hours?

[bikeman25]

Hello

Started this Robocopy file transfer last night of a folder on This Desktop to 256GB USB 3.2 Gen 1 Flash drive---Still transferring at this time even,  is that normal?    first time ever using Robocopy, previously just used WIndows 11 File Transfer Dialog and let it go for 24-48 hours when i updated this little flash drive,  but decided to try Robocopy, but still seems slow to me

Only security on system is Windows Defender, and Malwarebytes Premium

I guess this little Sandisk flash drive is good,  would take a heck of a long time to fully test it, but i can if need be

 

Transferring Bunch of Small Purchased Music Files--108GB worth

WIth Windows File transfer was moving at 3.42MB/s or lower

Robocopy not sure if it was really any faster lol, as its still not done, and started this at 6:30PM last night

Flash drive is Sandisk Dual USB 3.2 Gen 1 A & C (For Laptop use)   

Purchased From Amazon arond February 21st

 

Edited September 18, 2023 by bikeman25

[+virtorio MVC]

What command did you use? You can set the number of threads using the /mt parameter.

Even with all the overhead of transferring a bunch of files, that seems excessively slow.

[bikeman25]

Here is the Command i used

 "F:\Music Backup"/S /E /Z /ZB /R:5 /W:5 /TBD /NP /V /MT:16

And yes it does seem slow to me, thought would be a little faster this way, but i guess not lol, unless i entered something wrong with the command or this Sandisk flash drive stinks, or usb 3.0 hub adds extra slowness

 

[i_was_here]

You could always benchmark the drive, connect it to different USB ports, and benchmark again to see if the port is the issue.

[bikeman25]

Ok Then will try that

As this Powershell Robocopy keeps going around and around,  it looked earlier like it was on last directory in the folder, and then restarted again.    

don't see option to cancel it,  so i guess my options are let  it continue, or maybe restart system if it keeps going around and around,  i swear i seen this directory in the list earlier 

Been going since 6:30PM Last night lol

 

This drive is Sandisk Duo USB A/C dual drive 256GB,  Purchased February 21st For my birthday to keep copy of the Purchased Music Folder,  Size of that Group 108GB after combining all Former Xbox/Groove Music Purchases when i had Windows Phone years ago/plus the 30 free albums back then (yes i grabbed them all lol) 

Plus Music Bought from Amazon over the years

First time ever used Robocopy method lol, thought by morning it would've been done today lol

 

 

Edited September 19, 2023 by bikeman25

[goretsky Supervisor]

Hello,

Some thoughts, in no particular order:

• The USB flash drive's DRAM cache could be exhausted, meaning that slower NAND Flash RAM dies are being written to directly, instead of being buffered through the cache.
• The USB flash drive might be running hot as a result of the continuous, sustained writes, resulting in it slowing down to prevent any sort of thermal damage from occurring.
• Hardware issue with the USB flash drive
• File system corruption issue with the USB flash drive

In instances of #2, where I noticed a USB flash drive was getting hot, I put the drive into one of my dishwashing gloves, then put the  drive in the center of an ice-filled cup.  This seemed to cool things down enough to let copies work at expected speeds.

Regards,

Aryeh Goretsky
 

[mcloum]

On 18/09/2023 at 23:37, bikeman25 said:

Here is the Command i used

 "F:\Music Backup"/S /E /Z /ZB /R:5 /W:5 /TBD /NP /V /MT:16

And yes it does seem slow to me, thought would be a little faster this way, but i guess not lol, unless i entered something wrong with the command or this Sandisk flash drive stinks, or usb 3.0 hub adds extra slowness

 

I think your Robocopy command is a little more complex than it needs to be. If you're just doing a straight 1:1 copy from destination to source then do something like

robocopy <source> <destination> /MIR /R:1 W:1 /NP /MT:x /LOG:<path to a log files>

change 'x' in /MT to a suitable threads for your CPU.

 

Also as already stated your USB might be the limiting factor or the source drive your copying from.

 

 

[bikeman25]

Yeah Source Drive is Western Digital 4TB Black Hard drive

Destination Disk is or was suppose to be Sandisk 256GB Flash drive (wanted extra backup copy of the Important Music files on that flash drive, but its very very slow to copy to either one of the Flash drives thus far--i gave up trying to get the files on it for now, but i may try again maybe later tonight and see how it does with this updated RoboCopy command that is less complex, maybe i'll have better luck 

 

[+BudMan MVC]

just because your stick is 3.2 doesn't mean what your plugging it into is that..  Did you ever do the benchmark on the stick?

I just looked in my draw and pulled out a 3.1 32GB stick and ran a winsat on it..  What is your disk showing?

C:\Windows\system32>winsat disk -drive h
Windows System Assessment Tool
> Running: Feature Enumeration ''
> Run Time 00:00:00.00
> Running: Storage Assessment '-drive h -ran -read'
> Run Time 00:00:01.09
> Running: Storage Assessment '-drive h -seq -read'
> Run Time 00:00:01.69
> Running: Storage Assessment '-drive h -seq -write'
> Run Time 00:01:15.64
> Running: Storage Assessment '-drive h -flush -seq'
> Run Time 00:00:14.86
> Running: Storage Assessment '-drive h -flush -ran'
> Run Time 00:00:12.45
> Dshow Video Encode Time                      0.00000 s
> Dshow Video Decode Time                      0.00000 s
> Media Foundation Decode Time                 0.00000 s
> Disk  Random 16.0 Read                       27.14 MB/s          6.1
> Disk  Sequential 64.0 Read                   112.99 MB/s          6.8
> Disk  Sequential 64.0 Write                  50.26 MB/s          5.3
> Average Read Time with Sequential Writes     7.050 ms          5.3
> Latency: 95th Percentile                     5.068 ms          6.6
> Latency: Maximum                             5345.888 ms          1.5
> Average Read Time with Random Writes         5.835 ms          5.7
> Total Run Time 00:01:46.02

C:\Windows\system32>

So to simulate what your doing, since I only have 32GB stick, and want it to finish quicker I copied over 13.5 worth of music to it.. This is what I got using this robocopy

robocopy c:\test\music h:\ /mir /R:1 /W:1 /NP /MT

              Total    Copied   Skipped  Mismatch    FAILED    Extras   
   Dirs :       130       130         1         0         0         6   
  Files :      1407      1407         0         0         0         9   
  Bytes :  13.576 g  13.576 g         0         0         0   732.2 k   
  Times :   7:55:07   0:47:34                       0:00:00   0:06:03   
                                                                        
                                                                        
  Speed :             5106509 Bytes/sec.                                
  Speed :             292.196 MegaBytes/min.                            
  Ended : Tuesday, December 5, 2023 5:11:40 PM                          

So what 47 minutes to copy over 1407 files in 130 different dir for 13.5GB.. Can see the speed worked out to about 5MBps - so even if you were seeing that speed.. 108GB should take you what approx. about 6 hours..

So you got something wrong, either in reading your files or writing to your stick.. Or you getting some really bad speed writing too it..

I think the /MT can mess with the times that it shows.. I believe /mt without calling out specific number defaults to 8, but that might of changed haven't used robocopy in years.  So 47:34 is close but by clock, I know I started this at 4:18 and finished at 5:11:40 so that works to like 53 minutes total..

robocopy c:\test\music h:\ /mir /R:1 /W:1 /NP /MT

-------------------------------------------------------------------------------
   ROBOCOPY     ::     Robust File Copy for Windows
-------------------------------------------------------------------------------

  Started : Tuesday, December 5, 2023 4:18:02 PM
   Source : c:\test\music\
     Dest = h:\

    Files : *.*

  Options : *.* /S /E /DCOPY:DA /COPY:DAT /PURGE /MIR /NP /MT:8 /R:1 /W:1

 

[bikeman25]

Haven't benchmarked the stick yet, but i can shortly here

 

My Folder of the Music is stored on the 4TB Storage Drive--WD Black 4TB 

Windows 11 Pro clean installed less than 2 weeks ago, so gonna try the stick asap here again, and benchmark it to see what i get as soon as possible here

Hopefully some good results

USB Ports on the Rear of the PC is USB 3.2 Gen 1 i believe

Eventually i may get brave enough and invest in a Total SSD System, and just use the 4TB WD Black Drive as Extra External backup.   but as of right now still may main storage drive

 

*Though haven't run chkdsk on the 4TB Drive in ages, but i assume it should be fine i think lol*   

 

Edited December 6, 2023 by bikeman25
More Info

[bikeman25]

Results of Winsat on the Flash drive

C:\Windows\System32>winsat disk -drive f
Windows System Assessment Tool
> Running: Feature Enumeration ''
> Run Time 00:00:00.00
> Running: Storage Assessment '-drive f -ran -read'
> Run Time 00:00:01.44
> Running: Storage Assessment '-drive f -seq -read'
> Run Time 00:00:04.61
> Running: Storage Assessment '-drive f -seq -write'
> Run Time 00:13:25.05
> Running: Storage Assessment '-drive f -flush -seq'
> Run Time 00:00:56.25
> Running: Storage Assessment '-drive f -flush -ran'
> Run Time 00:00:54.22
> Dshow Video Encode Time                      0.00000 s
> Dshow Video Decode Time                      0.00000 s
> Media Foundation Decode Time                 0.00000 s
> Disk  Random 16.0 Read                       16.65 MB/s          5.8
> Disk  Sequential 64.0 Read                   104.74 MB/s          6.6
> Disk  Sequential 64.0 Write                  0.64 MB/s          2.1
> Average Read Time with Sequential Writes     26.628 ms          1.9
> Latency: 95th Percentile                     67.113 ms          1.9
> Latency: Maximum                             300.513 ms          6.9
> Average Read Time with Random Writes         25.919 ms          1.9
> Total Run Time 00:15:22.06

 

[Ixion]

That write speed sucks, suspect that's your answer right there!

[spaceelf]

You can try without the hub, and check for newer USB controller drivers or drive firmwares if there are any available.  That's a bit strange.

Could just be overheated in some instances.  I generally prefer SSDs to flash media for large transfers but interesting.

[bikeman25]

Second Sandisk 256GB Flash Drive Results

Sandisk dual USB A on one side, and USB C on other side drive

 

C:\Windows\System32>winsat disk -drive f
Windows System Assessment Tool
> Running: Feature Enumeration ''
> Run Time 00:00:00.00
> Running: Storage Assessment '-drive f -ran -read'
> Run Time 00:00:01.27
> Running: Storage Assessment '-drive f -seq -read'
> Run Time 00:00:05.03
> Running: Storage Assessment '-drive f -seq -write'
> Run Time 00:03:06.38
> Running: Storage Assessment '-drive f -flush -seq'
> Run Time 00:00:34.59
> Running: Storage Assessment '-drive f -flush -ran'
> Run Time 00:00:53.28
> Dshow Video Encode Time                      0.00000 s
> Dshow Video Decode Time                      0.00000 s
> Media Foundation Decode Time                 0.00000 s
> Disk  Random 16.0 Read                       15.56 MB/s          5.7
> Disk  Sequential 64.0 Read                   88.31 MB/s          6.3
> Disk  Sequential 64.0 Write                  3.29 MB/s          2.3
> Average Read Time with Sequential Writes     16.173 ms          3.2
> Latency: 95th Percentile                     42.170 ms          1.9
> Latency: Maximum                             6020.096 ms          1.4
> Average Read Time with Random Writes         25.412 ms          1.9
> Total Run Time 00:04:42.02

 

Both have terrible write speed though for some reason,  i may have to invest at some point in a Solid State Based External for first time--other external drives have i'll list below this comment

Western Digital Elements 500GB Desktop (2003) (First External, USB 2.0)

Western Digital Passport 2TB Ultra (Blue)

Western Digital Passport 2TB (RED)

Seagate 1TB   Portable USB 3.0

Western Digital in Enclosure 1TB WD Black in Ugreen USB 3.2 Gen 1 enclosure

Seagate 8TB  in Sabrent Enclosure  **Won this Drive  a CES Twitter Contest January 2019**   Originally the 4TB WD Black was gonna be the Backup Drive,   this was Primary til last year when i received Western Digital My Book 8TB to Become Primary, as Seagate 8TB turned out to be SMR Drive* 

Western Digital 8TB My Book (Primary Backup drive) 

*Note Sabrent USB Hub was cheap when i bought it,  gonna try at Christmas to afford a better one hopefully,   right now it doesn't function too well even though Powered USB Hub if i attempt to run 2 of the Externals with there own Power or even the bus powered Passport drives at same time*    

 

Internally on Desktop have the following

Samsung 970 Evo Plus 500GB Boot NvMe  *almost bought a Western Digital NvMe,  but decided in Christmas of 2020 to go for the Samsung instead--my first Non Western Digital,  my Previous AMD based FX 8310 system i had Western Digital Blue Sata 250GB Boot Drive, but then i only stored WIndows and a few small programs, and made sure rest of files was stored on another drive)   

Samsung 860 Evo 1TB Game SSD

Western Digital WD Black 4TB Storage drive

 

I just recently clean installed Windows 11 Pro x64 bit on this Desktop,  far as i know drivers are up to date as can be i think lol.        So maybe just bad luck on those drive purchases i suppose

 

Edited December 6, 2023 by bikeman25

[+BudMan MVC]

Yeah those speeds blow!! something is not right.. Wasn't there some issue with san disks from a while back? I remember some hubub about them but didn't pay it much mind because I don't have any..  Most of the ones I have bought in recent years have been SP ( Silicon Power).. But yeah with those write speeds no wonder it not finishing.

I knew there was some issues with san disks..

https://arstechnica.com/gadgets/2023/08/sandisk-extreme-ssds-are-worthless-multiple-lawsuits-against-wd-say/

But that is about SSD disks, not usb sticks from quick read over.

So you have any other sticks you could try winsat on?

edit: or do you have another machine you could test these in, even if not 3.2 usb port.

 

 

 

 

[bikeman25]

Do have a Secondary Gaming Laptop i can test these 2 Drives on

(Laptop has USB 3.0 ports) don't think it's Gen 2 or 1 on that machine, and 1 USB C port)

Laptop is HP Omen 15_ce019dx (purchased December 2017 )

I have another 128GB Sandisk i think it is (Really gotta stop buying Sandisk Flash drives lol)  and another 64GB Western Digital branded one,   1 Corasir 16GB i believe, and 

Will post the results once i finish the testing of those on other PC asap here

**These drives was both tested with a Rear USB 3.2 Gen 1 Port,  USB Hub was not used at all during this**  1 at a time, and nothing else was connected over to the system at all during that time**  

 

Same Flash Drive Sandisk Dual  USB A and C on one--Tested on PC 2 Gaming Laptop

Microsoft Windows [Version 10.0.19045.3758]
(c) Microsoft Corporation. All rights reserved.

C:\Windows\system32>winsat disk -drive E
Windows System Assessment Tool
> Running: Feature Enumeration ''
> Run Time 00:00:00.00
> Running: Storage Assessment '-drive E -ran -read'
> Run Time 00:00:01.47
> Running: Storage Assessment '-drive E -seq -read'
> Run Time 00:00:05.77
> Running: Storage Assessment '-drive E -seq -write'
> Run Time 00:06:48.30
> Running: Storage Assessment '-drive E -flush -seq'
> Run Time 00:00:28.55
> Running: Storage Assessment '-drive E -flush -ran'
> Run Time 00:01:16.33
> Dshow Video Encode Time                      0.00000 s
> Dshow Video Decode Time                      0.00000 s
> Media Foundation Decode Time                 0.00000 s
> Disk  Random 16.0 Read                       14.29 MB/s          5.6
> Disk  Sequential 64.0 Read                   83.10 MB/s          6.2
> Disk  Sequential 64.0 Write                  1.54 MB/s          2.1
> Average Read Time with Sequential Writes     13.563 ms          3.7
> Latency: 95th Percentile                     138.841 ms          1.9
> Latency: Maximum                             644.054 ms          1.9
> Average Read Time with Random Writes         36.800 ms          1.9
> Total Run Time 00:08:41.27

C:\Windows\system32>

Edited December 7, 2023 by bikeman25

[binaryzero]

robocopy "<source>" "<destination>" /R:1 /E /mir /MT:8 /SEC /tee /log+"<path to log file>"

[+BudMan MVC]

So if you go here https://ssd-tester.com/

That drive doesn't benchmark very well.. But better than what your seeing..

if your looking for something that is fast for write - might be a good place to look.

As another example - I took one of my old 512GB nmve sticks that I Put in a case and these are the benchmarks. Same usb-c port that the usb stick used.. What a difference less than a minute for the same 13.5GB of data..  My CD drive bench is below as well.

-------------------------------------------------------------------------------
   ROBOCOPY     ::     Robust File Copy for Windows
-------------------------------------------------------------------------------

  Started : Wednesday, December 6, 2023 9:56:01 PM
   Source : c:\test\music\
     Dest : h:\

    Files : *.*

  Options : *.* /S /E /DCOPY:DA /COPY:DAT /PURGE /MIR /NP /MT:8 /R:1 /W:1

------------------------------------------------------------------------------

               Total    Copied   Skipped  Mismatch    FAILED    Extras
    Dirs :       130       130         1         0         0         1
   Files :      1407      1407         0         0         0         0
   Bytes :  13.576 g  13.576 g         0         0         0         0
   Times :   0:06:07   0:00:36                       0:00:00   0:00:04


   Speed :           399097747 Bytes/sec.
   Speed :           22836.556 MegaBytes/min.
   Ended : Wednesday, December 6, 2023 9:56:42 PM

Here is the winsat on it

C:\Windows\system32>winsat disk -drive h
Windows System Assessment Tool
> Running: Feature Enumeration ''
> Run Time 00:00:00.00
> Running: Storage Assessment '-drive h -ran -read'
> Run Time 00:00:00.55
> Running: Storage Assessment '-drive h -seq -read'
> Run Time 00:00:02.59
> Running: Storage Assessment '-drive h -seq -write'
> Run Time 00:00:02.33
> Running: Storage Assessment '-drive h -flush -seq'
> Run Time 00:00:00.63
> Running: Storage Assessment '-drive h -flush -ran'
> Run Time 00:00:00.61
> Dshow Video Encode Time                      0.00000 s
> Dshow Video Decode Time                      0.00000 s
> Media Foundation Decode Time                 0.00000 s
> Disk  Random 16.0 Read                       198.76 MB/s          7.7
> Disk  Sequential 64.0 Read                   375.55 MB/s          8.0
> Disk  Sequential 64.0 Write                  390.00 MB/s          8.0
> Average Read Time with Sequential Writes     0.217 ms          8.6
> Latency: 95th Percentile                     0.508 ms          8.7
> Latency: Maximum                             1.282 ms          8.9
> Average Read Time with Random Writes         0.211 ms          8.9
> Total Run Time 00:00:06.83

C:\Windows\system32>

C:\Windows\system32>winsat disk -drive C
Windows System Assessment Tool
> Running: Feature Enumeration ''
> Run Time 00:00:00.00
> Running: Storage Assessment '-drive C -ran -read'
> Run Time 00:00:00.14
> Running: Storage Assessment '-drive C -seq -read'
> Run Time 00:00:02.14
> Running: Storage Assessment '-drive C -seq -write'
> Run Time 00:00:01.61
> Running: Storage Assessment '-drive C -flush -seq'
> Run Time 00:00:00.33
> Running: Storage Assessment '-drive C -flush -ran'
> Run Time 00:00:00.34
> Dshow Video Encode Time                      0.00000 s
> Dshow Video Decode Time                      0.00000 s
> Media Foundation Decode Time                 0.00000 s
> Disk  Random 16.0 Read                       807.28 MB/s          8.6
> Disk  Sequential 64.0 Read                   876.08 MB/s          8.4
> Disk  Sequential 64.0 Write                  835.89 MB/s          8.3
> Average Read Time with Sequential Writes     0.074 ms          8.9
> Latency: 95th Percentile                     0.150 ms          8.9
> Latency: Maximum                             0.324 ms          8.9
> Average Read Time with Random Writes         0.065 ms          8.9
> Total Run Time 00:00:04.66

C:\Windows\system32>

If your goal is backup and "fast" prob wouldn't use usb sticks.. But even the 3.1 usb interface seems to be pretty fast..  That benchmark site listed to shows a few really fast sticks..

[bikeman25]

Yeah my goal is kinda fast backup and yeah think time to get away from USB Sticks

Much better than what I've been seeing for sure

I'll get it all sorted in due time etiher for Christmas or Birthday in February,   as  if i tried to backup even a Smallish directory with these usb sticks i'd be waiting for a while i imagine lol.        The one time i did a direct File copy,  started at  11pm, and didn't finish til 11am next day lol

 

Of Course my Samsung 970 Evo Plus 500GB M.2 NvMe was silently replaced in August 2021 with a newer version, with a newer Controller---(this ones has had no firmwire updates at all--Old Phoenix controller on this one)  Debated if i was worth replacing or not, then i decided well probably not lol

 

So kinda will look into something a wee faster, doesn't have to be lightning speed really, just enough to properly backup the data.     Do have a Cloud Copy of the Purchased Music Folder which contains former XBox/Groove Music Free albums i got when had WIndows Phone back in 2014-2015 time frame (don't think those are replaceable hence why i got multiple backups on several drives as well lol)  Amazon bought songs,  Older CD Rips from many years ago lol,  

Edited December 7, 2023 by bikeman25
More Info

[Rigby]

I wouldn't trust USB sticks for backups anyway. Besides being horribly slow they really aren't meant for long term storage and they can die without warning.

[Ixion]

This was a plain old USB 3.1 Hard drive

[bikeman25]

Of Course in June 2020 This Motherboard was the Cheapest 10th Gen Compatible board my local PC shop had available in order to upgrade to Intel 10th Gen  10700 CPU go along with my new case and Video card at the time

I gotta benchmark my WD My Passport Red 2TB drive

either my USB 3.2 Gen 1 hub is slow as molasses--plan on purchasing a better usb hub with Christmas Gift card, and hopefully improves drive speed a little bit 

*Why Heck do the Power USB hub lose communication with a Powered USB Enclosure with WD 1TB Drive inside it, and therefore have to redo transfer of updated Folder to WD My Passport Drive*  This happens all the time whenever i attempt to use 2 Devices even with the Hub sighs

 

*When i upgrade to either 14th Gen in the Future or 15th Gen Intel or Maybe back to AMD in future,  not cheaping out on following items-- Motherboard,  CPU getting decent one, that i can hopefully still cool with air Cooling,  very nervous on AIOs leaking lol still.

And if go with Intel Processor again, then Nvidia video card,  if go with AMD for first time since AMD FX  then AMD video card (thats way i always been lol)     

 Upgrading Power Supply with a Real Good Quality one (may do that prior to next system refresh, maybe even for Birthday in February lol--Current Power Supply EVGA G3 650 Watt been in use since February 2018, of course back then i had more room to work on my Computers--since moved since then and less room and more family annoying me whenever i'm working on any type of upgrade*

 

Edited December 8, 2023 by bikeman25