85 posts in this topic

Posted

Fast copy software for internal network traffics

Urgently required

Hello

Urgently for business ... looking for the most fastest software for data copy or data transfer in internal network between computers in the same network

I've goggled and found TeraCopy and RichCopy to be the most fastest tools around

When I test both of them I've found it is maximum 9:10 MB/second ... which is really slow for our needs

Please advise for a tool free or paid that could be as fast as possible?

Note: lan is about 20 node ... and daily data traffics in between is about 150:200 GB/day ... Yes we do work in that volume in a daily base ... so please advise for a suitable one?

Also all computers running windows 7 and XP

Share this post


Link to post
Share on other sites

Posted

well you are about at the limit of 100Mb/s, unless you have gigabit you aren't going to see any faster.

100Mb/s is roughly 10MB/s...9.1MB/s is right at the limitation of a 100Mb/s line. You will never see 100% of 100Mb/s 80-90% is normally what you would see, and again you are right there with 9.1. Look at your network graph in task man.

Share this post


Link to post
Share on other sites

Posted

well you are about at the limit of 100Mb/s, unless you have gigabit you aren't going to see any faster.

100Mb/s is roughly 10MB/s...9.1MB/s is right at the limitation of a 100Mb/s line. You will never see 100% of 100Mb/s 80-90% is normally what you would see, and again you are right there with 9.1. Look at your network graph in task man.

Hmm..I thought 100mbps ethernet connected to a 100mbps switch can only do half-duplex. You can only get full-duplex if you go computer-to-computer over ethernet. But I haven't really messed with networking stuff for awhile.

As far as what is the fastest transfer method, I believe something like FTP will smoke SMB (Windows file sharing) in overall throughput. But Windows file sharing is so much easier to work with...

You will need to describe the workflow situation a little bit more. Are you just looking to transfer files over the network constantly? Or are you working with files from across the network but ultimately want them to exist where they are and not be transferred to your machine?

When I test both of them I've found it is maximum 9:10 MB/second ... which is really slow for our needs

Didn't see that. Yeah, as sc203 pointed out 9.10MB/s is about as fast as you are going to get on your standard 100mbps network. You will need to upgrade to gigabit or 10 gigabit network to do better. If you are wired with Cat5e or better you can do 1 gigabit (so long as your wire distances are <100 meter). To do 10 gigabit I think you will have to re-wire to Cat6.

Share this post


Link to post
Share on other sites

Posted

LOL. Get Fiber.... or 10GBE

Lets think about it. your average hard drive can read/write around what, 60-80MB/sec maybe..MAX. 60-80MBx8 bits/byte = 480-640Mbit/sec. IF you get Gigabit, you're probably going to be capped by the drive, though I've never come close to actual Gbit speeds, the fastest i've seen is around 700-800Mbit. Teracopy/xcopy whatever won't speed things up. If you want the fastest transfers possible, you need a good RAID array or SSD's, and a fast pipe..10GBE or fiber

Share this post


Link to post
Share on other sites

Posted

No you can have full duplex on a switch.

FTP is a fast protocol, however when transferring with multiple streams it blows FTP away, this is not windows sharing anything and it is as similar as to FTP to windows file sharing.

Share this post


Link to post
Share on other sites

Posted

LOL. Get Fiber.... or 10GBE

Lets think about it. your average hard drive can read/write around what, 60-80MB/sec maybe..MAX. 60-80MBx8 bits/byte = 480-640Mbit/sec. IF you get Gigabit, you're probably going to be capped by the drive, though I've never come close to actual Gbit speeds, the fastest i've seen is around 700-800Mbit. Teracopy/xcopy whatever won't speed things up. If you want the fastest transfers possible, you need a good RAID array or SSD's, and a fast pipe..10GBE or fiber

Gbit you will only see 800-900Mb/s tops That whole you will never see more than 90% of its capability comes into play here as well. Roughly that is 80-90MB/s.

Share this post


Link to post
Share on other sites

Posted

Urgently for business ... looking for the most fastest software for data copy or data transfer in internal network between computers in the same network

Doesn't all of this "copy" or "file management" software just use the standard Windows API's like everything else? This isn't really a software issue as stated above, it's hardware.

Software isn't going to automagically make your hardware run faster.

Share this post


Link to post
Share on other sites

Posted

Doesn't all of this "copy" or "file management" software just use the standard Windows API's like everything else? This isn't really a software issue as stated above, it's hardware.

Software isn't going to automagically make your hardware run faster.

yes, it doesn't matter what you use software wise. But even still you are never going to see 100% of gig, 100, or 10Ge.

Share this post


Link to post
Share on other sites

Posted

well you are about at the limit of 100Mb/s, unless you have gigabit you aren't going to see any faster.

100Mb/s is roughly 10MB/s...9.1MB/s is right at the limitation of a 100Mb/s line. You will never see 100% of 100Mb/s 80-90% is normally what you would see, and again you are right there with 9.1. Look at your network graph in task man.

Would you be able to get a bit more performance by using UDP rather than TCP, while sacrificing data reliability?

Share this post


Link to post
Share on other sites

Posted

Is that 200GB all new files or just updated files?

Maybe some software can "delta" copy the new bits instead of rewritting all of files all over again ?

Share this post


Link to post
Share on other sites

Posted

Would you be able to get a bit more performance by using UDP rather than TCP, while sacrificing data reliability?

Is the bit faster worth it? And it depends, it may not be faster at all it may actually be slower. doing multistreamed copies is what is going to maximize bandwidth and over look drive speed for the most part, upd and tcp aside.

multistreamed copies copy multiple files at the same time. even though the drive may be opening or closing one file another, or many, is/are still in the copy data mode. during a standard copy, a file is opened, copied to, then closed. it takes longer to do a 1 for 1 copy than a multistreamed copy, while the file is being opened and closed during a 1 for 1 copy no data is being transferred, so the network is not being utilized at this point. Multistreamed copies is more effective for many smaller files than large files.

Share this post


Link to post
Share on other sites

Posted

Is that 200GB all new files or just updated files?

Maybe some software can "delta" copy the new bits instead of rewritting all of files all over again ?

Deduplication would be able to do this, but you need the master copy first.

Share this post


Link to post
Share on other sites

Posted

Is the bit faster worth it? And it depends, it may not be faster at all.

Isn't the actual data portion the same? (1500 bytes IIRC)

Yes the packet is smaller, but def. not worth it.

@xendrome - I guess these apps, even if they used the standard windows APIs, could compress the data where possible to make it transfer the overall file quicker, even if you have the same throughput.

Original file: 100MB, compressed by software, 90MB. The latter would transfer quicker. You couldn't compress it too heavily in real time though without a powerful processor, and you'd need to do the same on the opposite end (uncompress) so I'm not sure it's worth all the effort.

Share this post


Link to post
Share on other sites

Posted

I think what you are looking for is rsync. Runs on Linux and Windows.

Share this post


Link to post
Share on other sites

Posted

I would HIGHLY suggest if your working with that size of data 150-200GB a day that you really move to gig.

Gig switches are only a couple dollars more than 10/100 these days. If not cheaper.. if you computers are newer, its quite possible they have 10/100/1000 nics already. So all you need is the switch. Even if your computers don't have gig, cards can be very reasonable as well.. Just picked up a gig pci express card for $35 and that was a highly rated intel card.. You can pick up cheaper ones for less than $20 for sure.

As stated going gig is going to drastically increase your file copies.. I see 90MBps quite often just using crappy cheap disks..

Here is a pull of a file from my home server (vm running on esxi 5 even)


C:\test>robocopy z:\test c:\test test1.avi

-------------------------------------------------------------------------------

   ROBOCOPY	 ::	 Robust File Copy for Windows

-------------------------------------------------------------------------------

  Started : Fri Mar 30 12:23:12 2012

   Source : z:\test\

	 Dest : c:\test\


	Files : test1.avi

  Options : /COPY:DAT /R:1000000 /W:30

------------------------------------------------------------------------------

						   1	z:\test\

100%		New File			 699.9 m		test1.avi

------------------------------------------------------------------------------

			   Total	Copied   Skipped  Mismatch	FAILED	Extras

	Dirs :		 1		 0		 1		 0		 0		 0

   Files :		 1		 1		 0		 0		 0		 0

   Bytes :  699.92 m  699.92 m		 0		 0		 0		 0

   Times :   0:00:07   0:00:07					   0:00:00   0:00:00


   Speed :			95501024 Bytes/sec.

   Speed :			5464.612 MegaBytes/min.


   Ended : Fri Mar 30 12:23:20 2012

95MBps --- that is just screaming - and 10x what your seeing! So yeah the small cost of moving to gig would be well worth it if you move a bunch of data. I could never in a million years go back to 10/100 speeds! Now pushing files might be a bit slower

------------------------------------------------------------------------------


			   Total	Copied   Skipped  Mismatch	FAILED	Extras

	Dirs :		 1		 0		 1		 0		 0		 0

   Files :		 1		 1		 0		 0		 0		 0

   Bytes :  699.92 m  699.92 m		 0		 0		 0		 0

   Times :   0:00:10   0:00:10					   0:00:00   0:00:00



   Speed :			69553200 Bytes/sec.

   Speed :			3979.866 MegaBytes/min.


   Ended : Fri Mar 30 12:27:44 2012

But 70MBps is nothing to sneeze at either - and would be night and day for you would it not?

Also the suggestion of rsync might be a good idea as well, if your just making changes to files - rsync can just push the differences vs having to push/pull the whole size of the file. But that would entail a bit of training and setup to use for sure.

Share this post


Link to post
Share on other sites

Posted

Thanks a lot for all the feedback.

I am sorry for not placing enough information about the exact situation.

We are a small media producing company that are working under short term contracts out sourced for some local TV channels to record its shows on a daily base and do video encoding to various format like .avi, .rm, .flv, .mp3

Share this post


Link to post
Share on other sites

Posted

You must understand something very very clearly. You are at the physical limit of the 10/100 switch. You need to change this out to get faster transfer rates. If these are new files that are getting created this large there is no software out there that will help your situation. If these files are being added to, eg: you have a word file and each day new data is getting added to that word file, then there is software that can help you.

Switch out the 10/100 switch for a gigabit switch to increase bandwidth utilization 10 times. The 10/100 switch is what is slowing you down, all equipment has to be 1000Mb/s for you to get 1000Mb/s, if not it will slow down to the slowest piece of equipment

-------------------------------------------------

Example

1pc 10/100/1000 - 1 switch 10/100 - 1pc 10/100/1000

the switch can only handle up to 100 so all traffic going through this will be at 100

--------------------------------------------------

1pc 10/100 - 1 switch 10/100/1000 - 1pc 10/100/1000

The pc to the right will get gigabit, which is all great, but will only trasfer from the pc on the left at 100 speed.

--------------------------------------------------

I hope that clears things up for you.

1 person likes this

Share this post


Link to post
Share on other sites

Posted

You must understand something very very clearly. You are at the physical limit of the 10/100 switch. You need to change this out to get faster transfer rates. If these are new files that are getting created this large there is no software out there that will help your situation. If these files are being added to, eg: you have a word file and each day new data is getting added to that word file, then there is software that can help you.

Switch out the 10/100 switch for a gigabit switch to increase bandwidth utilization 10 times. The 10/100 switch is what is slowing you down, all equipment has to be 1000Mb/s for you to get 1000Mb/s, if not it will slow down to the slowest piece of equipment.

Ok, I will buy 4 switches cisco small business edition 8 ports each 10-100-1000 (then bridge each 2 switches together to have a 14 ports one because the 16 ports or 24 ports one is very expensive) and a cat 6 cables.

So you mean the transfer with the same software (teracopy 0 richcopy) will jump up when I use the 10/100/1000 instead?

The current rate is 9:10 mb/second as I mentioned earlier.

What is the expected rate with the 10/100/1000 tools (lan card or port - switch - cable)?

Thanks

Share this post


Link to post
Share on other sites

Posted

I just gave you an example of what speed you can expect.. I see like 70MBps easy, with 95MBps in some testing over my home gig network. This is using cheap cards and cheap disks - everything is on the cheap ;)

Keep in mind as sc302 clearly pointed out -- you have to have gig end to end. There can not be anything between the 2 machines at 100 or your going to be limited to that speed.

Once you move to gig, your bottle neck is more likely going to be your drives and not the network.. Currently your network is limiting you to that 9.1MBps speed.. This is about the best you can see with 100mbit.. Yah might be able to squeeze out 10 -- but just doing the simple math. Saying you could get all 100mbps -- which you can't!!! Simple 100/8 give you max theory raw speed of like 12.5, now take into account the overhead of tcp and then the protocol your using to move the files.. And yeah 10 is freaking SCREAMING!!! on a 100mbps connection.. If you seeing 9.1 that is pretty good.

But once you move to gig you should see like 50 without any problem. If not -- let us know and we can do some digging to why not, etc.

BTW you don't really need cat6 - 5e will work just fine! But you can get 16 or 24 port gigs for pretty cheap vs buying multiple little ones.

Here is a 24 porter for $150

http://www.newegg.co...N82E16817111031

Gig switches are not that exp these days. Since your moving large files, once you could go gig you could look into jumbo frames as well.. This could get you some extra speed.. But need to make sure your cards and switches support it with the same sizes. Quite often the issues you run into with jumbo is that cards don't support same sizes that the switches do, etc. Or different makers cards might not support the same sizes as the cards in the other machines, etc.

Shoot here is a 24 porter for $107

http://www.newegg.co...N82E16833704065

Share this post


Link to post
Share on other sites

Posted

what budman said!!

Share this post


Link to post
Share on other sites

Posted

.

Share this post


Link to post
Share on other sites

Posted

The answer is bittorrent.

That has as much to do with this as cows and sheep do. Might as well have told him that the answer is a big mac with a side of fries. But I am in for some schooling....how is that the answer on a lan?

Share this post


Link to post
Share on other sites

Posted

torrents? No - sorry not by any means at all.

Now if he had to move these large files to like 500 different machines on the same lan ok. But he is moving files from the recording boxes to the transcoding boxes. There are only a few boxes involved. Torrents don't make a lot of sense for that sort of thing.

Shoot once he goes gig he should be able to move these files in couple of minutes vs the like 20 it takes him now.

Share this post


Link to post
Share on other sites

Posted

Really thanks a lot for all these information from all of you, really too much appreciated.

I think the 10/100/1000 or Giga is a must now.

Please anotherthing but I do not know if it is suitable to ask here or open another new topic, it is regarding the switch:-

  • a) Does connecting two switches, each one is 8 ports which equal 16-2=14 ports works the same as one switch with 16 ports or still one switch with 16 ports is a better performance in my cae?
  • b) When I am comparing switches with the same features except different branding ... does it differ to use cisco or d-link or tp-link ...etc ?

Thanks a lot

Share this post


Link to post
Share on other sites

Posted

Well, I've contacted my co-workers and they got 3 switches from this model:

http://www.newegg.com/Product/Product.aspx?Item=N82E16833150080&Tpk=sg%20100d-08

to connect two of them for a big one.

and a one switch from model:

http://h10010.www1.hp.com/wwpc/ie/en/sm/WF06b/12883-12883-4172267-4172280-4172280-4220258-4220261.html?dnr=1

to connect it with the one from te three above to have a big one.

plus 20 pci cards d-link 10/100/1000

so please advise ... shall we use these devices as they already purchased (really they got it very fast lol) or it is better to get another alternate?

thanks a lot and really very sorry for bothering you all

also regarding the pci lan cards ... is it better to use the motherboard built-in lan port ? or to use the pci device ? I mean regarding performance and copy or transfer performance and time consuming?

Thanks

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0

  • Recently Browsing   0 members

    No registered users viewing this page.