Strange Throughputs...

[Attie]

Hi all,

 

I've got a Gigabit network, and I'm currently transferring large files between some of the computers.

The attached diagram (diagram.png) will give you an idea of the network layout (there are other machines, but these are all that are relevant for now).

 

A) is a Dell laptop, very new, running Win 8.1. it has a 100Mb/s Ethernet port (I was very surprised that it wasn't Gbit)

B) is a custom workstation, high-spec server hardware (that is now a few generations old), running Win 7 64-bit

C) is a Linux (Ubuntu Server 14.04) VM, running on ESXi. This also has a server-grade NIC, and an i7.

 

I've just been pushing some large files around, and noticed something odd happening. I'm using Samba (SMB/CIFS) in all cases, and TeraCopy (which I highly recommend).

While copying from B to C I usually see around 40MB/s throughput, which I consider to be perfectly adequate.

 

However, I've just set off another large file transfer from A to B. This is limited by the 100MB/s NIC on the laptop, and the 11MB/s that it's achieving is to be expected.

The interesting thing, is that when I started this transfer (from A to B), the transfer that was already running (from B to C) dropped to a steady 6.5 MB/s!

 

TeraCopy has the option to Pause transfers, and I captured the network throughput stats from Resource Monitor during a brief pause (res_mon.png).

When paused, that first transfer shoots back up to its previous 38-40MB/s.

 

Can anyone make any sense of this?

Perhaps the throughput of that GBit switch is really not very good? (They are both Netgear GS308, with -iirc- a claimed max. throughput of 16Gb/s).

 

I look forward to reading some hypotheses!

 

Attie

[+BudMan MVC]

"TeraCopy (which I highly recommend)."

 

Well that is not something I would agree with or recommend at all - all the testing I have done with that software is does nothing but actually slow down copy operations... Its pointless software!!  I would recommend you remove it and test.

 

Here is the thing, 40MBps is not all that good to start with for a gig network - its ok, not bad.. But it is not good.. Here is a push to my VM nas (win7 x64bit) running on a n40l hp microserver with nothing for specs, with $40 nic on the esxi host.  And built in nic my OLD i5 dell 8300.

 

 

For starters I would suggest what you can do on the wire, maybe your disks are the bottleneck - you say large files - this was 1.2GB file - are you moving lots of little files vs 1 large one that add up to a large xfer?

 

What is your actual wire speed from B to C taking the disks and file copy protocol out of the picture - what does say iperf show?  I just posted up a copy of 3.0.9 for windows in another thread couple weeks back.  https://www.neowin.net/forum/topic/1234695-iperf-309-windows-build/ You can grab it for your linux sure its out there already.  So here is testing from my workstation to my nas.

 

Work is the client.

 

Nas is the client

 

As you can see over 900mbps each way..  Which works out to 900/8 = 112MBps without any overhead, no disk slow downs, etc..  40*8 = 320mbps, which is not very good for gig network to be honest.  But sure it could be your switch, could be your samba config?  Could be lots of things.. So quick test lets see what you can get on your wire.  Then lets try and get that speed up to something respectable for a gig network ;)

 

So here is a quick copy of video from my nas to my workstation - large 2.6GB file, over 100MBps..  This is with budget hardware!!

 

 

Now my switch is a bit pricy for home budgets, $200 cisco sg300.. But I saw these speeds when running on a netgear gs108t as well.. But still within home budget range..

 

Here is where your run into an issue even after we get your speeds up to something more where it should be..  You should be seeing 50MBps write for sure, and over 70 read even with the lowest budget stuff..  If not better your still going to run into a problem with doing multiple operations at 2 different speeds.. Your trying to read from B and send it to C, while at the same time write to the disk on B from A at a much slower speed of 100mbps, which is more like 90ish..

 

Can you get your A connected at Gig?  This would help with overall speeds - but your still going to have issues doing read and write to the disk at the same time going to be a hit on your B system.

 

But lets work on your network wire speed first!! On gig lets try to get your iperf results over 700mbps for sure..

 

edit: Just for ref info

My workstation is 192.168.1.00, my VM acting as my storage/nas is storage.local.lan, 192.168.1.8.  My workstation has 2 mapped drives M and Z to the VM, M drive is where I store all my home video, and z is where I store other media and other files, etc. The workstation is an old dell XPS 8300, nic it came with.  The esxi host 5.5 build 2143827, nic is  HP 412648-B21 NC360T PCI-Express DP GigaBit Adapter which I picked up for like $40, its a dual port card.

 

Here is the vm on esxi that is my nas/storage box

 

I only gave it 1 cpu, and 2GB of ram ;) running with all those other vms on N40L, which cpu is AMD Turion II Neo N40L 1.5GHz 2-Core.  Your not talking rocketship hardware by any sense of the imagination ;)

[Attie]

Hi BudMan,

 

Thanks for your reply.

I agree that 40MB/s isn't great, but it's acceptable for me at the moment - something to look into later perhaps.

 

The wire-speeds are great - proof from iperf below.

 

A Server, B Client (remember, 100Mb/s)

 

B Server, A Client? (remember, 100Mb/s)

 

B Server, C Client

 

C Server, B Client

 

 

I went a step further, and ran iperf simultaneously, replicating the loads that the file copies produced.

C Server, B Client (green bar shows period where B Server, A Client was running)

 

B Server, A Client (run during the above test)

 

 

I also tried using robocopy (a tool I've not heard of until this week funnily enough - I'm more comfortable in a Linux environment)

The following shot shows three states:

Red underline - Copy from B to A

Blue underline - Copy from B to A and Copy from C to B

Green underline - Copy from C to B (aborted copy from B to A)

Blue underline - Copy from B to A and Copy from C to B (restarted copy from B to A again)

 

edit: These iperf tests prove that the switch isn't the issue, and that even robocopy grinds the throughput right down.

edit2: Apologies, I forgot to mention that no, these are large files (1GB+), not many small files - I'm aware of the overheads involved there.

 

 

Thanks for voicing your opinion about TeraCopy - I'll steer clear for a while and see how things go.

I started using it a while ago due to the way the Windows UI handles issues (broadly speaking, it doesn't).

[+BudMan MVC]

Well your iperf clearly shows your wire speeds are rocking!!!  You using ipv6 link local addresses in that one connection even ;)  Out of curiosity do you have ipv6 actually setup or do your windows host just default with all that teredo and 6to4 and isatap, etc..  Do you have ipv6 internet?  If not i would prob turn all the local ipv6 off.

 

Your problem is most likely disk based, or protocol issues - you should be seeing way more than 40MBps with those wire speeds.

 

Are you using smb, smb2 or smb3?  Or actually cifs?  I would think your really doing smb - but I don't know what flavor of samba your running on your linux box?  Also what are you using for your visualization?  What disks are you sporting?

 

Clearly your wire speed is screaming - so why only 40MBps from B to C??  Or C to B do you see the same - lets forget the red headed step child talking at 100mbps until we find out what is slowing you down to 40MBps when your wire is an autobahn ;)

[Attie]

Hey,

 

Sorry for the delay - I wanted to post some I/O results as well, but had to go out...

 

I'm not consciously using IPv6 (neither DHCP or my ISP are setup for it), but Windows is doing it's usual playing and toying with my world (Grr).

I've disabled it in the NIC's properties page (see below). Would you recommend anything further? (i.e: http://tweaks.com/windows/40099/how-to-properly-disable-ipv6/)

 

 

I've just run HD Tune on a couple of my workstation's disks (there are more, but I think/hope they are unimportant for this investigation).

It has an 840 Pro for the OS drive: the throughput is limited due to the SATA II interface, but the latency and IPOS are tasty - this does appear to be running at roughly SATA I (1.5Gb/s) speeds though... I'll investigate that, but it isn't the bottleneck

 

It also has a 8 disk RAID 5 on an Adaptec 5805 (with BBU), the array size is mis-reported here: (this is the source and destination for the transfers in the previous tests)

 

Is there any software that you'd recommend that can benchmark reads / writes via the filesystem (rather than direct to disk)?

I remember using CrystalMark (??) in the past, but the installer I just downloaded had nasties bundled in so I kicked it out.

 

 

The VM is running on ESXi. It has two virtual disks:

One, mounted on / (root) is stored on another 840 Pro and is indented to be largely read-only. (aside from system updates, etc)

The second is mounted on /home and is stored on a 4TB WD Red disk (WD40EFRX) - this was the source for the C -> B transfer in the previous tests.

I've just done some simple dd tests, below:

I'd comment on the poor write performance of the SSD, but I'm not sure what ESXi is doing in the middle (or even how well SSDs are supported - TRIM?), and read is good.

As you can see the filesystem I/O on /home/test is good. Doing I/O on the 'raw' (virtualized) disk gets ~130MB/s reads.

 

 

 

In terms of Windows smb/smb2/smb3 I'm not sure...  :blush:

I would guess that it fell back to 2.1 between A and B (B being Win7)

 

On the VM I'm running 'Samba version 4.1.6-Ubuntu', and the man page states 'Default: client max protocol = SMB3' (I've not consciously overridden it).

 

edit: I've tried running 'Get-SmbConnection' in PowerShell, but I get 'The term 'Get-SmbConnection' is not recognized as the name of a cmdlet, function, script file, or operable program.'

edit2: It would appear that I'm using SMB2... is WireShark seriously the easiest way to determine this?!

 

 

Thanks for your help!

Attie

[+BudMan MVC]

yeah most likely wireshark going to be the easy way to validate the smb protocol in use..

I am out of town for work, just checking in on stuff real quick so didn't have time to read your post in detail. But from your benchmarks on your tests, and your wire speeds it seems very odd that your only seeing 40MBps -- you should be seeing way higher than that for sure!

Not sure if going to get time for any good replies until next week or evening

[Attie]

Hi again,

 

Apologies for the silence - I wanted to give you a chance to get home again - thanks for your help with this...

 

Embarrassingly, in the meantime I've deduced that the array in my workstation is seeing fantastic read speeds (~590MB/s), but the filesystem writes are very poor (~45MB/s).

 

Here are some results: SSD (on workstation), RAID (on workstation), SMB (on ESXi VM)

• RAID -> SMB   TeraCopy: ~102MB/s    robocopy: ~104MB/s
• SMB -> RAID   TeraCopy: ~43MB/s      robocopy: ~42MB/s

• RAID -> SSD    TeraCopy: ~135MB/s    robocopy: ~1GB/s (yeah right...)
• SSD -> RAID    TeraCopy: ~ 44MB/s     robocopy: ~1GB/s (ahem...) || ~254MB/s after flushing the cache || avg. ~60MB/s (zoom zoom to 70-90%, then it get's sticky)... hmm...

• SSD -> SMB     TeraCopy: ~103MB/s    robocopy: ~104MB/s
• SMB -> SSD     TeraCopy: ~58MB/s      robocopy: ~89MB/s

As you can see, TeraCopy and robocopy are getting similar transfer speeds between the RAID and the SMB share.

Things got funky when I started comparing between volumes on the workstation.

I turn TeraCopy's 'Use system write cache' option off as matter of course, so that once TeraCopy has completed, the file should be on the disk. (I'll typically use TeraCopy for moving large sets of files, and the Windows UI for everything else).

 

I can only presume here that the 48GB of RAM on my workstation may have something to do with the... insane... transfer speeds seen by robocopy.

A couple of tests from RAID -> SMB were even performed with the array idle / spun down (I got distracted)... and even after trying to flush the cache.  :blink:

The SSD and RAID both have Windows Write-caching disabled (the RAID controller has it's own cache), which adds to the confusion over excessive speeds seen by robocopy.

 

 

In summary I think this concludes it, I need to take a look at my array.

The 5805 only has 512MB of RAM/cache, and I was shifting a ~1.5GB file here, so it's on-board cache wasn't providing all that data.

 

Attie

[Attie]

I've just run iozone on the array, with the following command:

iozone -f /cygdrive/l/test/iotest -b /cygdrive/c/iozone.xls -a -s 1G -y 512 -q 8192 -I -p

• -f /cygdrive/l/test/iotest - use a file on the array - L:\test\iotest
• -b /cygdrive/c/iozone.xls - save results to C:\iozone.xls
• -a - use the automatic set of tests (some overrides below)
• -s 1G - filesize of 1GB (larger than the controller's cache)
• -y 512 - minimum record size (512 kB)
• -q 8192 - maximum record size (8MB)
• -I - use direct I/O (should avoid the cache)
• -p - purge the processor cache before each operation

 

... and generated this graph from the results (which I'm not too sure I trust fully...)

 

This indicates write speeds ranging from ~890MB/s to ~1.6GB/s.

That would indicate a write speed for each drive of ~230MB/s (1.6GB / 7 = 230MB), which is unlikely seeing as the array is made up of nothing special - 8x Seagate ST31500341AS drives.

The datasheet claims a "Sustained Data Rate" of 120MB/s, which makes the 890MB/s result almost believable (~127MB/s each).

The array was also churning away during the tests, which implies that the Direct I/O flag was having the desired effect...

 

To add to the confusion, this indicates that the array is not the problem. I wonder if I'm missing something here...

 

Attie