Fast copy software for internal network traffics

[NoUserName]

Fast copy software for internal network traffics

Urgently required

Hello

Urgently for business ... looking for the most fastest software for data copy or data transfer in internal network between computers in the same network

I've goggled and found TeraCopy and RichCopy to be the most fastest tools around

When I test both of them I've found it is maximum 9:10 MB/second ... which is really slow for our needs

Please advise for a tool free or paid that could be as fast as possible?

Note: lan is about 20 node ... and daily data traffics in between is about 150:200 GB/day ... Yes we do work in that volume in a daily base ... so please advise for a suitable one?

Also all computers running windows 7 and XP

[sc302 Veteran]

well you are about at the limit of 100Mb/s, unless you have gigabit you aren't going to see any faster.

100Mb/s is roughly 10MB/s...9.1MB/s is right at the limitation of a 100Mb/s line. You will never see 100% of 100Mb/s 80-90% is normally what you would see, and again you are right there with 9.1. Look at your network graph in task man.

[Shadrack]

  On 30/03/2012 at 16:10, sc302 said:

well you are about at the limit of 100Mb/s, unless you have gigabit you aren't going to see any faster.

100Mb/s is roughly 10MB/s...9.1MB/s is right at the limitation of a 100Mb/s line. You will never see 100% of 100Mb/s 80-90% is normally what you would see, and again you are right there with 9.1. Look at your network graph in task man.

Hmm..I thought 100mbps ethernet connected to a 100mbps switch can only do half-duplex. You can only get full-duplex if you go computer-to-computer over ethernet. But I haven't really messed with networking stuff for awhile.

As far as what is the fastest transfer method, I believe something like FTP will smoke SMB (Windows file sharing) in overall throughput. But Windows file sharing is so much easier to work with...

You will need to describe the workflow situation a little bit more. Are you just looking to transfer files over the network constantly? Or are you working with files from across the network but ultimately want them to exist where they are and not be transferred to your machine?

  Quote

When I test both of them I've found it is maximum 9:10 MB/second ... which is really slow for our needs

Didn't see that. Yeah, as sc203 pointed out 9.10MB/s is about as fast as you are going to get on your standard 100mbps network. You will need to upgrade to gigabit or 10 gigabit network to do better. If you are wired with Cat5e or better you can do 1 gigabit (so long as your wire distances are <100 meter). To do 10 gigabit I think you will have to re-wire to Cat6.

[SirEvan]

LOL. Get Fiber.... or 10GBE

Lets think about it. your average hard drive can read/write around what, 60-80MB/sec maybe..MAX. 60-80MBx8 bits/byte = 480-640Mbit/sec. IF you get Gigabit, you're probably going to be capped by the drive, though I've never come close to actual Gbit speeds, the fastest i've seen is around 700-800Mbit. Teracopy/xcopy whatever won't speed things up. If you want the fastest transfers possible, you need a good RAID array or SSD's, and a fast pipe..10GBE or fiber

[sc302 Veteran]

No you can have full duplex on a switch.

FTP is a fast protocol, however when transferring with multiple streams it blows FTP away, this is not windows sharing anything and it is as similar as to FTP to windows file sharing.

[sc302 Veteran]

  On 30/03/2012 at 16:29, SirEvan said:

LOL. Get Fiber.... or 10GBE

Lets think about it. your average hard drive can read/write around what, 60-80MB/sec maybe..MAX. 60-80MBx8 bits/byte = 480-640Mbit/sec. IF you get Gigabit, you're probably going to be capped by the drive, though I've never come close to actual Gbit speeds, the fastest i've seen is around 700-800Mbit. Teracopy/xcopy whatever won't speed things up. If you want the fastest transfers possible, you need a good RAID array or SSD's, and a fast pipe..10GBE or fiber

Gbit you will only see 800-900Mb/s tops That whole you will never see more than 90% of its capability comes into play here as well. Roughly that is 80-90MB/s.

[xendrome]

  On 30/03/2012 at 15:58, NoUserName said:

Urgently for business ... looking for the most fastest software for data copy or data transfer in internal network between computers in the same network

Doesn't all of this "copy" or "file management" software just use the standard Windows API's like everything else? This isn't really a software issue as stated above, it's hardware.

Software isn't going to automagically make your hardware run faster.

[sc302 Veteran]

  On 30/03/2012 at 16:38, xendrome said:

Doesn't all of this "copy" or "file management" software just use the standard Windows API's like everything else? This isn't really a software issue as stated above, it's hardware.

Software isn't going to automagically make your hardware run faster.

yes, it doesn't matter what you use software wise. But even still you are never going to see 100% of gig, 100, or 10Ge.

[CentralDogma]

  On 30/03/2012 at 16:10, sc302 said:

well you are about at the limit of 100Mb/s, unless you have gigabit you aren't going to see any faster.

100Mb/s is roughly 10MB/s...9.1MB/s is right at the limitation of a 100Mb/s line. You will never see 100% of 100Mb/s 80-90% is normally what you would see, and again you are right there with 9.1. Look at your network graph in task man.

Would you be able to get a bit more performance by using UDP rather than TCP, while sacrificing data reliability?

[IceBlackIce]

Is that 200GB all new files or just updated files?

Maybe some software can "delta" copy the new bits instead of rewritting all of files all over again ?

[sc302 Veteran]

  On 30/03/2012 at 16:43, CentralDogma said:

Would you be able to get a bit more performance by using UDP rather than TCP, while sacrificing data reliability?

Is the bit faster worth it? And it depends, it may not be faster at all it may actually be slower. doing multistreamed copies is what is going to maximize bandwidth and over look drive speed for the most part, upd and tcp aside.

multistreamed copies copy multiple files at the same time. even though the drive may be opening or closing one file another, or many, is/are still in the copy data mode. during a standard copy, a file is opened, copied to, then closed. it takes longer to do a 1 for 1 copy than a multistreamed copy, while the file is being opened and closed during a 1 for 1 copy no data is being transferred, so the network is not being utilized at this point. Multistreamed copies is more effective for many smaller files than large files.

[sc302 Veteran]

  On 30/03/2012 at 16:44, IceBlackIce said:

Is that 200GB all new files or just updated files?

Maybe some software can "delta" copy the new bits instead of rewritting all of files all over again ?

Deduplication would be able to do this, but you need the master copy first.

[cybertimber2008]

  On 30/03/2012 at 16:44, sc302 said:

Is the bit faster worth it? And it depends, it may not be faster at all.

Isn't the actual data portion the same? (1500 bytes IIRC)

Yes the packet is smaller, but def. not worth it.

@xendrome - I guess these apps, even if they used the standard windows APIs, could compress the data where possible to make it transfer the overall file quicker, even if you have the same throughput.

Original file: 100MB, compressed by software, 90MB. The latter would transfer quicker. You couldn't compress it too heavily in real time though without a powerful processor, and you'd need to do the same on the opposite end (uncompress) so I'm not sure it's worth all the effort.

[Mr Winkle]

I think what you are looking for is rsync. Runs on Linux and Windows.

[+BudMan MVC]

I would HIGHLY suggest if your working with that size of data 150-200GB a day that you really move to gig.

Gig switches are only a couple dollars more than 10/100 these days. If not cheaper.. if you computers are newer, its quite possible they have 10/100/1000 nics already. So all you need is the switch. Even if your computers don't have gig, cards can be very reasonable as well.. Just picked up a gig pci express card for $35 and that was a highly rated intel card.. You can pick up cheaper ones for less than $20 for sure.

As stated going gig is going to drastically increase your file copies.. I see 90MBps quite often just using crappy cheap disks..

Here is a pull of a file from my home server (vm running on esxi 5 even)

C:\test>robocopy z:\test c:\test test1.avi
-------------------------------------------------------------------------------
   ROBOCOPY	 ::	 Robust File Copy for Windows
-------------------------------------------------------------------------------
  Started : Fri Mar 30 12:23:12 2012
   Source : z:\test\
	 Dest : c:\test\

	Files : test1.avi
  Options : /COPY:DAT /R:1000000 /W:30
------------------------------------------------------------------------------
						   1	z:\test\
100%		New File			 699.9 m		test1.avi
------------------------------------------------------------------------------
			   Total	Copied   Skipped  Mismatch	FAILED	Extras
	Dirs :		 1		 0		 1		 0		 0		 0
   Files :		 1		 1		 0		 0		 0		 0
   Bytes :  699.92 m  699.92 m		 0		 0		 0		 0
   Times :   0:00:07   0:00:07					   0:00:00   0:00:00

   Speed :			95501024 Bytes/sec.
   Speed :			5464.612 MegaBytes/min.

   Ended : Fri Mar 30 12:23:20 2012

95MBps --- that is just screaming - and 10x what your seeing! So yeah the small cost of moving to gig would be well worth it if you move a bunch of data. I could never in a million years go back to 10/100 speeds!

Now pushing files might be a bit slower

------------------------------------------------------------------------------

			   Total	Copied   Skipped  Mismatch	FAILED	Extras
	Dirs :		 1		 0		 1		 0		 0		 0
   Files :		 1		 1		 0		 0		 0		 0
   Bytes :  699.92 m  699.92 m		 0		 0		 0		 0
   Times :   0:00:10   0:00:10					   0:00:00   0:00:00


   Speed :			69553200 Bytes/sec.
   Speed :			3979.866 MegaBytes/min.

   Ended : Fri Mar 30 12:27:44 2012

But 70MBps is nothing to sneeze at either - and would be night and day for you would it not?

Also the suggestion of rsync might be a good idea as well, if your just making changes to files - rsync can just push the differences vs having to push/pull the whole size of the file. But that would entail a bit of training and setup to use for sure.

[NoUserName]

Thanks a lot for all the feedback.

I am sorry for not placing enough information about the exact situation.

We are a small media producing company that are working under short term contracts out sourced for some local TV channels to record its shows on a daily base and do video encoding to various format like .avi, .rm, .flv, .mp3 ? and upload it to famous websites plus a .mpeg copy to be given to the channel itself.

Exact situation simply like this:

- 20 computers DELL OptiPlex GX620 Pentium d using windows XP, all computers' cpu placed in a one big rack, no monitor, no speakers, no mouse, no keyboards, but just the cpu and lan and power cables plus the LNB cable.

- Each DELL OptiPlex GX620 one has a satellite pci card (TeVii S420 DVB-S PCI) as shown here:

http://www.tevii.com...ucts_S420_1.asp

Each computer do nothing except recording ? this is why the low configurations for OptiPlex GX620 working great for us.

Each computer recording a very big file which is about 10-11 gb on a daily base (each new day mean a new file).

Also 4 other main computers using windows 7with i7 2600k with 16 gb ram and high features to do the video encoding.

To save a place we only have the cpu of the OptiPlex GX620 and access it from the i7s computers using remote desktop.

Sorry for spamming or bothering you with my silly details but it is just to place you in my place with exact situation.

Back to topic subject ? the need for the copy software is to copy the .mpeg2 recorded files from the DELL OptiPlex GX620 computers to the i7 ones to do the main work which is video encoding.

I've realized that dell network is 10\100\1000 ? but right now I am only on 10\100 mode only ? (Thanks for pointing out the 10/100/1000 tip).

Also switches we have is only 10/100 ? Thanks again for this tip guys ;)

So the issue now is a magic solution or a tool to do the copy as fast as possible.

Also using 10/100/1000 switches and cables will help improve the speed or may still the same? Please clear this issue.

I think we have 5 or maybe more D-Link 10-100-1000 lan pci cards to may be used if this will help (this to be used with i7 computers but I think the motherboard gigabyte z68 maybe support 10-100-1000 mode.

So please advise what I shall do to do a fast copy and save our time as it is a very critical issue for our contract that may lead to cancel it if we did not succeed it.

Thanks for your time and sorry once again for a long post.

[sc302 Veteran]

You must understand something very very clearly. You are at the physical limit of the 10/100 switch. You need to change this out to get faster transfer rates. If these are new files that are getting created this large there is no software out there that will help your situation. If these files are being added to, eg: you have a word file and each day new data is getting added to that word file, then there is software that can help you.

Switch out the 10/100 switch for a gigabit switch to increase bandwidth utilization 10 times. The 10/100 switch is what is slowing you down, all equipment has to be 1000Mb/s for you to get 1000Mb/s, if not it will slow down to the slowest piece of equipment

-------------------------------------------------

Example

1pc 10/100/1000 - 1 switch 10/100 - 1pc 10/100/1000

the switch can only handle up to 100 so all traffic going through this will be at 100

--------------------------------------------------

1pc 10/100 - 1 switch 10/100/1000 - 1pc 10/100/1000

The pc to the right will get gigabit, which is all great, but will only trasfer from the pc on the left at 100 speed.

--------------------------------------------------

I hope that clears things up for you.

[NoUserName]

  On 30/03/2012 at 18:49, sc302 said:

You must understand something very very clearly. You are at the physical limit of the 10/100 switch. You need to change this out to get faster transfer rates. If these are new files that are getting created this large there is no software out there that will help your situation. If these files are being added to, eg: you have a word file and each day new data is getting added to that word file, then there is software that can help you.

Switch out the 10/100 switch for a gigabit switch to increase bandwidth utilization 10 times. The 10/100 switch is what is slowing you down, all equipment has to be 1000Mb/s for you to get 1000Mb/s, if not it will slow down to the slowest piece of equipment.

Ok, I will buy 4 switches cisco small business edition 8 ports each 10-100-1000 (then bridge each 2 switches together to have a 14 ports one because the 16 ports or 24 ports one is very expensive) and a cat 6 cables.

So you mean the transfer with the same software (teracopy 0 richcopy) will jump up when I use the 10/100/1000 instead?

The current rate is 9:10 mb/second as I mentioned earlier.

What is the expected rate with the 10/100/1000 tools (lan card or port - switch - cable)?

Thanks

[+BudMan MVC]

I just gave you an example of what speed you can expect.. I see like 70MBps easy, with 95MBps in some testing over my home gig network. This is using cheap cards and cheap disks - everything is on the cheap ;)

Keep in mind as sc302 clearly pointed out -- you have to have gig end to end. There can not be anything between the 2 machines at 100 or your going to be limited to that speed.

Once you move to gig, your bottle neck is more likely going to be your drives and not the network.. Currently your network is limiting you to that 9.1MBps speed.. This is about the best you can see with 100mbit.. Yah might be able to squeeze out 10 -- but just doing the simple math. Saying you could get all 100mbps -- which you can't!!! Simple 100/8 give you max theory raw speed of like 12.5, now take into account the overhead of tcp and then the protocol your using to move the files.. And yeah 10 is freaking SCREAMING!!! on a 100mbps connection.. If you seeing 9.1 that is pretty good.

But once you move to gig you should see like 50 without any problem. If not -- let us know and we can do some digging to why not, etc.

BTW you don't really need cat6 - 5e will work just fine! But you can get 16 or 24 port gigs for pretty cheap vs buying multiple little ones.

Here is a 24 porter for $150

http://www.newegg.co...N82E16817111031

Gig switches are not that exp these days. Since your moving large files, once you could go gig you could look into jumbo frames as well.. This could get you some extra speed.. But need to make sure your cards and switches support it with the same sizes. Quite often the issues you run into with jumbo is that cards don't support same sizes that the switches do, etc. Or different makers cards might not support the same sizes as the cards in the other machines, etc.

Shoot here is a 24 porter for $107

http://www.newegg.co...N82E16833704065

[sc302 Veteran]

what budman said!!

[+Cryton Subscriber²]

.

[sc302 Veteran]

  On 30/03/2012 at 20:01, Cryton said:

The answer is bittorrent.

That has as much to do with this as cows and sheep do. Might as well have told him that the answer is a big mac with a side of fries. But I am in for some schooling....how is that the answer on a lan?

[+BudMan MVC]

torrents? No - sorry not by any means at all.

Now if he had to move these large files to like 500 different machines on the same lan ok. But he is moving files from the recording boxes to the transcoding boxes. There are only a few boxes involved. Torrents don't make a lot of sense for that sort of thing.

Shoot once he goes gig he should be able to move these files in couple of minutes vs the like 20 it takes him now.

[NoUserName]

Really thanks a lot for all these information from all of you, really too much appreciated.

I think the 10/100/1000 or Giga is a must now.

Please anotherthing but I do not know if it is suitable to ask here or open another new topic, it is regarding the switch:-

• a) Does connecting two switches, each one is 8 ports which equal 16-2=14 ports works the same as one switch with 16 ports or still one switch with 16 ports is a better performance in my cae?
  
• b) When I am comparing switches with the same features except different branding ... does it differ to use cisco or d-link or tp-link ...etc ?
  

Thanks a lot

[NoUserName]

Well, I've contacted my co-workers and they got 3 switches from this model:

http://www.newegg.com/Product/Product.aspx?Item=N82E16833150080&Tpk=sg%20100d-08

to connect two of them for a big one.

and a one switch from model:

http://h10010.www1.hp.com/wwpc/ie/en/sm/WF06b/12883-12883-4172267-4172280-4172280-4220258-4220261.html?dnr=1

to connect it with the one from te three above to have a big one.

plus 20 pci cards d-link 10/100/1000

so please advise ... shall we use these devices as they already purchased (really they got it very fast lol) or it is better to get another alternate?

thanks a lot and really very sorry for bothering you all

also regarding the pci lan cards ... is it better to use the motherboard built-in lan port ? or to use the pci device ? I mean regarding performance and copy or transfer performance and time consuming?

Thanks

[NoUserName]

also I forget to ask about the buffer size I should ask about for the switch and does it matter or not?

thanks