Jump to content



Photo

Fast copy software for internal network traffics

network data copy win7 winxp lan gigabit business backup

  • Please log in to reply
84 replies to this topic

#1 NoUserName

NoUserName

    No Title

  • Joined: 10-May 09
  • Location: No

Posted 30 March 2012 - 15:58

Fast copy software for internal network traffics
Urgently required
Hello
Urgently for business ... looking for the most fastest software for data copy or data transfer in internal network between computers in the same network
I've goggled and found TeraCopy and RichCopy to be the most fastest tools around
When I test both of them I've found it is maximum 9:10 MB/second ... which is really slow for our needs
Please advise for a tool free or paid that could be as fast as possible?
Note: lan is about 20 node ... and daily data traffics in between is about 150:200 GB/day ... Yes we do work in that volume in a daily base ... so please advise for a suitable one?
Also all computers running windows 7 and XP


#2 sc302

sc302

    Neowinian Senior

  • Tech Issues Solved: 23
  • Joined: 12-July 05
  • Location: NJ, USA

Posted 30 March 2012 - 16:10

well you are about at the limit of 100Mb/s, unless you have gigabit you aren't going to see any faster.

100Mb/s is roughly 10MB/s...9.1MB/s is right at the limitation of a 100Mb/s line. You will never see 100% of 100Mb/s 80-90% is normally what you would see, and again you are right there with 9.1. Look at your network graph in task man.

#3 Shadrack

Shadrack

    Neowinian Senior

  • Tech Issues Solved: 3
  • Joined: 20-December 01

Posted 30 March 2012 - 16:24

well you are about at the limit of 100Mb/s, unless you have gigabit you aren't going to see any faster.

100Mb/s is roughly 10MB/s...9.1MB/s is right at the limitation of a 100Mb/s line. You will never see 100% of 100Mb/s 80-90% is normally what you would see, and again you are right there with 9.1. Look at your network graph in task man.


Hmm..I thought 100mbps ethernet connected to a 100mbps switch can only do half-duplex. You can only get full-duplex if you go computer-to-computer over ethernet. But I haven't really messed with networking stuff for awhile.

As far as what is the fastest transfer method, I believe something like FTP will smoke SMB (Windows file sharing) in overall throughput. But Windows file sharing is so much easier to work with...

You will need to describe the workflow situation a little bit more. Are you just looking to transfer files over the network constantly? Or are you working with files from across the network but ultimately want them to exist where they are and not be transferred to your machine?

When I test both of them I've found it is maximum 9:10 MB/second ... which is really slow for our needs


Didn't see that. Yeah, as sc203 pointed out 9.10MB/s is about as fast as you are going to get on your standard 100mbps network. You will need to upgrade to gigabit or 10 gigabit network to do better. If you are wired with Cat5e or better you can do 1 gigabit (so long as your wire distances are <100 meter). To do 10 gigabit I think you will have to re-wire to Cat6.

#4 SirEvan

SirEvan

    Neowinian Senior

  • Joined: 17-April 03
  • Location: Santa Clara, CA
  • OS: Windows 8
  • Phone: HTC One (AT&T)

Posted 30 March 2012 - 16:29

LOL. Get Fiber.... or 10GBE

Lets think about it. your average hard drive can read/write around what, 60-80MB/sec maybe..MAX. 60-80MBx8 bits/byte = 480-640Mbit/sec. IF you get Gigabit, you're probably going to be capped by the drive, though I've never come close to actual Gbit speeds, the fastest i've seen is around 700-800Mbit. Teracopy/xcopy whatever won't speed things up. If you want the fastest transfers possible, you need a good RAID array or SSD's, and a fast pipe..10GBE or fiber

#5 sc302

sc302

    Neowinian Senior

  • Tech Issues Solved: 23
  • Joined: 12-July 05
  • Location: NJ, USA

Posted 30 March 2012 - 16:30

No you can have full duplex on a switch.

FTP is a fast protocol, however when transferring with multiple streams it blows FTP away, this is not windows sharing anything and it is as similar as to FTP to windows file sharing.

#6 sc302

sc302

    Neowinian Senior

  • Tech Issues Solved: 23
  • Joined: 12-July 05
  • Location: NJ, USA

Posted 30 March 2012 - 16:31

LOL. Get Fiber.... or 10GBE

Lets think about it. your average hard drive can read/write around what, 60-80MB/sec maybe..MAX. 60-80MBx8 bits/byte = 480-640Mbit/sec. IF you get Gigabit, you're probably going to be capped by the drive, though I've never come close to actual Gbit speeds, the fastest i've seen is around 700-800Mbit. Teracopy/xcopy whatever won't speed things up. If you want the fastest transfers possible, you need a good RAID array or SSD's, and a fast pipe..10GBE or fiber


Gbit you will only see 800-900Mb/s tops That whole you will never see more than 90% of its capability comes into play here as well. Roughly that is 80-90MB/s.

#7 xendrome

xendrome

    In God We Trust; All Others We Monitor

  • Tech Issues Solved: 9
  • Joined: 05-December 01
  • OS: Windows 8.1 Pro x64

Posted 30 March 2012 - 16:38

Urgently for business ... looking for the most fastest software for data copy or data transfer in internal network between computers in the same network


Doesn't all of this "copy" or "file management" software just use the standard Windows API's like everything else? This isn't really a software issue as stated above, it's hardware.

Software isn't going to automagically make your hardware run faster.

#8 sc302

sc302

    Neowinian Senior

  • Tech Issues Solved: 23
  • Joined: 12-July 05
  • Location: NJ, USA

Posted 30 March 2012 - 16:40

Doesn't all of this "copy" or "file management" software just use the standard Windows API's like everything else? This isn't really a software issue as stated above, it's hardware.

Software isn't going to automagically make your hardware run faster.

yes, it doesn't matter what you use software wise. But even still you are never going to see 100% of gig, 100, or 10Ge.

#9 CentralDogma

CentralDogma

    Neowinian Senior

  • Joined: 29-February 08

Posted 30 March 2012 - 16:43

well you are about at the limit of 100Mb/s, unless you have gigabit you aren't going to see any faster.

100Mb/s is roughly 10MB/s...9.1MB/s is right at the limitation of a 100Mb/s line. You will never see 100% of 100Mb/s 80-90% is normally what you would see, and again you are right there with 9.1. Look at your network graph in task man.


Would you be able to get a bit more performance by using UDP rather than TCP, while sacrificing data reliability?

#10 IceBlackIce

IceBlackIce

    Neowinian

  • Joined: 10-November 03

Posted 30 March 2012 - 16:44

Is that 200GB all new files or just updated files?
Maybe some software can "delta" copy the new bits instead of rewritting all of files all over again ?

#11 sc302

sc302

    Neowinian Senior

  • Tech Issues Solved: 23
  • Joined: 12-July 05
  • Location: NJ, USA

Posted 30 March 2012 - 16:44

Would you be able to get a bit more performance by using UDP rather than TCP, while sacrificing data reliability?


Is the bit faster worth it? And it depends, it may not be faster at all it may actually be slower. doing multistreamed copies is what is going to maximize bandwidth and over look drive speed for the most part, upd and tcp aside.

multistreamed copies copy multiple files at the same time. even though the drive may be opening or closing one file another, or many, is/are still in the copy data mode. during a standard copy, a file is opened, copied to, then closed. it takes longer to do a 1 for 1 copy than a multistreamed copy, while the file is being opened and closed during a 1 for 1 copy no data is being transferred, so the network is not being utilized at this point. Multistreamed copies is more effective for many smaller files than large files.

#12 sc302

sc302

    Neowinian Senior

  • Tech Issues Solved: 23
  • Joined: 12-July 05
  • Location: NJ, USA

Posted 30 March 2012 - 16:45

Is that 200GB all new files or just updated files?
Maybe some software can "delta" copy the new bits instead of rewritting all of files all over again ?

Deduplication would be able to do this, but you need the master copy first.

#13 cybertimber2008

cybertimber2008

    Neowinian Senior

  • Joined: 02-December 08

Posted 30 March 2012 - 16:51

Is the bit faster worth it? And it depends, it may not be faster at all.

Isn't the actual data portion the same? (1500 bytes IIRC)
Yes the packet is smaller, but def. not worth it.

@xendrome - I guess these apps, even if they used the standard windows APIs, could compress the data where possible to make it transfer the overall file quicker, even if you have the same throughput.
Original file: 100MB, compressed by software, 90MB. The latter would transfer quicker. You couldn't compress it too heavily in real time though without a powerful processor, and you'd need to do the same on the opposite end (uncompress) so I'm not sure it's worth all the effort.

#14 Mr Winkle

Mr Winkle

    Neowinian

  • Joined: 17-November 02
  • Location: Norfolk, UK

Posted 30 March 2012 - 16:59

I think what you are looking for is rsync. Runs on Linux and Windows.

#15 +BudMan

BudMan

    Neowinian Senior

  • Tech Issues Solved: 86
  • Joined: 04-July 02
  • Location: Schaumburg, IL
  • OS: Win7, Vista, 2k3, 2k8, XP, Linux, FreeBSD, OSX, etc. etc.

Posted 30 March 2012 - 17:29

I would HIGHLY suggest if your working with that size of data 150-200GB a day that you really move to gig.

Gig switches are only a couple dollars more than 10/100 these days. If not cheaper.. if you computers are newer, its quite possible they have 10/100/1000 nics already. So all you need is the switch. Even if your computers don't have gig, cards can be very reasonable as well.. Just picked up a gig pci express card for $35 and that was a highly rated intel card.. You can pick up cheaper ones for less than $20 for sure.

As stated going gig is going to drastically increase your file copies.. I see 90MBps quite often just using crappy cheap disks..

Here is a pull of a file from my home server (vm running on esxi 5 even)

C:\test>robocopy z:\test c:\test test1.avi
-------------------------------------------------------------------------------
   ROBOCOPY	 ::	 Robust File Copy for Windows
-------------------------------------------------------------------------------
  Started : Fri Mar 30 12:23:12 2012
   Source : z:\test\
	 Dest : c:\test\

	Files : test1.avi
  Options : /COPY:DAT /R:1000000 /W:30
------------------------------------------------------------------------------
						   1	z:\test\
100%		New File			 699.9 m		test1.avi
------------------------------------------------------------------------------
			   Total	Copied   Skipped  Mismatch	FAILED	Extras
	Dirs :		 1		 0		 1		 0		 0		 0
   Files :		 1		 1		 0		 0		 0		 0
   Bytes :  699.92 m  699.92 m		 0		 0		 0		 0
   Times :   0:00:07   0:00:07					   0:00:00   0:00:00

   Speed :			95501024 Bytes/sec.
   Speed :			5464.612 MegaBytes/min.

   Ended : Fri Mar 30 12:23:20 2012

95MBps --- that is just screaming - and 10x what your seeing! So yeah the small cost of moving to gig would be well worth it if you move a bunch of data. I could never in a million years go back to 10/100 speeds!

Now pushing files might be a bit slower
------------------------------------------------------------------------------

			   Total	Copied   Skipped  Mismatch	FAILED	Extras
	Dirs :		 1		 0		 1		 0		 0		 0
   Files :		 1		 1		 0		 0		 0		 0
   Bytes :  699.92 m  699.92 m		 0		 0		 0		 0
   Times :   0:00:10   0:00:10					   0:00:00   0:00:00


   Speed :			69553200 Bytes/sec.
   Speed :			3979.866 MegaBytes/min.

   Ended : Fri Mar 30 12:27:44 2012

But 70MBps is nothing to sneeze at either - and would be night and day for you would it not?

Also the suggestion of rsync might be a good idea as well, if your just making changes to files - rsync can just push the differences vs having to push/pull the whole size of the file. But that would entail a bit of training and setup to use for sure.



Click here to login or here to register to remove this ad, it's free!