ESXi - N40L - Slow Network Speeds

[Lee Barratt]

Hi Guys,

I am running 2 VMS, WHS2011 and WinXP, both have vmxnet3 drivers installed. The XP network says its running at 1.4GB/s and the WHS network is running at 10GB/s.

When I transfer files from one VM to the other i am getting speeds of 2MB/s. Why is this so slow??

I have run iperf and get speeds of 102 Mbits/sec

whilst the file transfer is in progress CPU usage is around 10% on both VM machines, but when I run iperf, CPU usage maxs out at 100% on the XP VM and about 55-65% on the WHS2011 VM.

Is the CPU being at 100% throttling the network?

I am running one physical network card,

Can anyone shed some light on this? I am a newbie to VMs but not to computers :)

thanks!

[Guest]

If your using the vmxnet3 driver shouldn't it be showing 10GB?

*Edit*

Just checked my XP VM out and indeed it shows 1.4GB. Never noticed that before.

102Mbit is about right for a 1GB NIC no?

[+John Teacake MVC]

Those speeds (That are reported in Windows) are for the Virtual Adapters. Although its true that transferring speeds between VM's on the same does write directly between Memory and not hit the wire so to speak. The speed will only be 1Gbps on the Physical Adapter by the way. Have you applied all the latest patches to ESXi, I would highly recommend it. My CPU usage is always really low, Infact I am surprised at how efficient it is. I just need some more memory in my baby!

http://www.vmware.com/patchmgr/findPatch.portal

[Detection]

100% CPU is likely to bottleneck just about anything you try to do with your machine

[+John Teacake MVC]

SK[' timestamp=1357165637' post='595429082]

If your using the vmxnet3 driver shouldn't it be showing 10GB?

*Edit*

Just checked my XP VM out and indeed it shows 1.4GB. Never noticed that before.

102Mbit is about right for a 1GB NIC no?

It might be for XP that it shows 1.4Gbps, Don't have one to test it. I have only seen it at 10GBps. Budman could chime in though and tell us if this is the case.

Yes your right just over 100MB for 1Gbps.

[Guest]

Well great thread anyway as you highlighted there's a new patch from the 20th of Dec I need to install :)

[Lee Barratt]

thanks, will install the patches and see how i get on

[Lee Barratt]

Ive installed the patches and am still getting around 2Mb/s transfer speeds between VMs.

any other ideas? shouldnt i be getting better speeds than this?

[Guest]

Very much so. Can you try another OS?

Is it the same speed both ways? IE Client to Server and Server to Client?

[Lee Barratt]

I only have another installation of XP which i have tried with WHS and still getting the same results, unfortunatly i dont have another OS to try

the speed seems the same both ways

[+John Teacake MVC]

Yeah so if you have Two VMs running and copying between a file share you should be getting much faster speeds than that. Have you installed VMWare Tools on Both OS's. I assume you have if you are using the vmxnet NIC.

[Lee Barratt]

yes, have installed the latest patches and VMtools.

both VMs have one CPU allocated and each has 4Gb of memory allocated.

[+BudMan MVC]

So I don't have an XP vm currently - but I could fire one up for sure.

But between my nas vm and win7 vm I show this, btw 102mbps is SLOW if you showing connected at gig speeds.

C:\Windows\System32>iperf -c storage -w 256k
------------------------------------------------------------
Client connecting to storage, TCP port 5001
TCP window size:  256 KByte
------------------------------------------------------------
[276] local 192.168.1.210 port 49233 connected with 192.168.1.8 port 5001
[ ID] Interval	   Transfer	 Bandwidth
[276]  0.0-10.0 sec   599 MBytes   502 Mbits/sec

I should see if I can get that up a bit - my physical machines see much higher. 800Mbps should be seen on a gig network fairly easy.

So file copy from my nas vm to my win7 vm

So I am seeing 50MBytes per second in this test.

But curious to what disks your writing/reading from - are you just trying to copy a file from datastore drive of vm to another datastore drive on the other vm. My nas has raw access to the 3 drives it uses for storage. So the above tests were from one of those disks to the datastore drive of the win7 vm

Also curious --- you gave 4GB to each of your VMs, do you have more than 8 in your N40L?

So I did a test that I think more reflects what your doing more likely. And that is from datastore drive to datastore drive of each VM. And in that test I got only 23MBps

What else is your datastore disk doing at the time?? That performance is not going to be the best, your 2 different VMs are accessing the same physical disk. So yeah there is going to be some contention for time.

[Lee Barratt]

thanks, i have 10GB of memory in my N40L, an 8Gb and a 2Gb stick.

I have a drivepool in WHS, consisting of 2TB and 1.5TB WD Green drives

I am transferring files from the pool to the XP machine which has a second 2TB drive attached.

The VMs themselves were stored on the 250Gb drive, but I have moved them to another 2TB drive for now.

The ESXi is installed on a 8GB USB pen drive.

Thanks

[Lee Barratt]

a bit of progress maybe?

under the WHS2011 VM

If i "pull" a file from the xp machine to the WHS drivepool i get speeds of 13.5MB/s

if i "push" a file from the WHS drivepool to the xp machine i get speeds of 2MB/s

could that be a write issue with the XP HDD ? (E: 2TB)

[Guest]

There's been issues in the past where thin discs show a performance hit. Are your XP discs thin?

[Lee Barratt]

the XP and WHS vm are both thick provisioned

Also, if i transfer a file from a physical XP machine to WHS vw across a 1gb network, i get 23MB/s

[+BudMan MVC]

thats still pretty crappy for a gig network. If I do a iperf or file copy to a physical machine from the vm I get this

C:\test>robocopy \\storage\media\cleanup c:\test test.mp4

-------------------------------------------------------------------------------
   ROBOCOPY	 ::	 Robust File Copy for Windows
-------------------------------------------------------------------------------

  Started : Thu Jan 03 10:31:22 2013

   Source = \\storage\media\cleanup\
	 Dest : c:\test\

	Files : test.mp4

  Options : /COPY:DAT /R:1000000 /W:30

------------------------------------------------------------------------------

						   1	\\storage\media\cleanup\
100%		New File			 650.8 m		test.mp4

------------------------------------------------------------------------------

			   Total	Copied   Skipped  Mismatch	FAILED	Extras
	Dirs :		 1		 0		 1		 0		 0		 0
   Files :		 1		 1		 0		 0		 0		 0
   Bytes :  650.86 m  650.86 m		 0		 0		 0		 0
   Times :   0:00:09   0:00:09					   0:00:00   0:00:00


   Speed :			74150984 Bytes/sec.
   Speed :			4242.953 MegaBytes/min.

   Ended : Thu Jan 03 10:31:31 2013

C:\test>iperf -c storage -w 256k
------------------------------------------------------------
Client connecting to storage, TCP port 5001
TCP window size:  256 KByte
------------------------------------------------------------
[336] local 192.168.1.100 port 16077 connected with 192.168.1.8 port 5001
[ ID] Interval	   Transfer	 Bandwidth
[336]  0.0-10.0 sec  1012 MBytes   849 Mbits/sec

Notice the 800+ Mbits per second over gig.. And in the 70MBps for file pull from my VM.

Do an iperf test from your physical to your VM, using a higher window size, say -w 256k You can see example of this test in my above output. On the server set it up with iperf -s -w 256k

[Lee Barratt]

Ive set up the physical XP machine as server, with

WHS vm as a client i get transfer 353Mb, bandwidth 295MBit/sec

XP vm as a client i get transfer 11.1Mb, bandwidth 8.59MBit/sec

[+John Teacake MVC]

SK[' timestamp=1357229689' post='595430390]

There's been issues in the past where thin discs show a performance hit. Are your XP discs thin?

Damn you beat me too it! I was going to say that. How are they formatted?

[+BudMan MVC]

Do you have 2 physical boxes we can play with - 295Mbps is that your iperf test.. That is pretty crappy for a gig network. Did you use the larger window size for your test?

at 8.59 mbps -- is that over wireless or something? Are you reading writing from a USB disk or something.. That is just horrific performance.

[Lee Barratt]

Damn you beat me too it! I was going to say that. How are they formatted?

I added the drives under the vclient and formatted them via windows - is that right? am a novice with ESXi

Do you have 2 physical boxes we can play with - 295Mbps is that your iperf test.. That is pretty crappy for a gig network. Did you use the larger window size for your test?

at 8.59 mbps -- is that over wireless or something? Are you reading writing from a USB disk or something.. That is just horrific performance.

yes, those figures are are with iperf on a wired Gb network

I've just performed another iperf test

physcial XP machine set as server (1Gb network) - wired

physical win7 laptop set as client (100Mb network) - wired

and i get 87Mbits/sec

the laptop has only got a 100Mb network card, so thats the max on this network - this is for this test only.

[+BudMan MVC]

Well 87mbps on 100mbit connection is about right.. Best you could see on 100mbit is around mid 90's -- think best I have ever seen is 97 something. Never ever going to see actual speed of the connection.

But on gig you should be seeing closer to 800s vs your under 300 number.

So question for you on your VM whs - did you give whs raw access to the disks, or did you create vmfs on the disks that the OS has access to.

So for example on my storage box

Notice that the 3 disks I have given it to use in a drive pool the disks are raw mapped, and then above that you see the virtual disk (on the datastore drive) that is the vmfs for the OS installed too.

Here is a guide on how to do that

http://vm-help.com/esx40i/SATA_RDMs.php

So for example here is my maps

/vmfs/volumes/4f677e4a-81443006-5233-2c768aadf656/RDMs # ls -la

drwxr-xr-x 1 root root 1120 Dec 26 13:12 .

drwxr-xr-t 1 root root 5180 Dec 29 14:59 ..

-rw------- 1 root root 2000398934016 Apr 7 2012 rdm1-rdmp.vmdk

-rw------- 1 root root 521 Dec 26 13:12 rdm1.vmdk

-rw------- 1 root root 750156374016 Apr 16 2012 rdm2-rdmp.vmdk

-rw------- 1 root root 520 Dec 26 13:12 rdm2.vmdk

-rw------- 1 root root 750156374016 Apr 14 2012 rdm3-rdmp.vmdk

-rw------- 1 root root 520 Dec 26 13:12 rdm3.vmdk

/vmfs/volumes/4f677e4a-81443006-5233-2c768aadf656/RDMs #

So couple of things to deal with on your slow performance - why your iperf between vms is so slow. And then make sure that your drive access is as fast as it can be. How are attaching the disks to your VMs could be a limiting factor - and have to look into the speed between vms from a network as well. I do recall seeing above 1Gbps before on iperf between machines

yeah just looked up a old thread about N40L performance - and in a test there I was seeing 1.5Gbps between vms - have to take a look why only seeing 500mbps now

https://www.neowin.net/forum/topic/1071115-esxi-hp-microserver-n40l-performance/page__view__findpost__p__594826393

So I got something not quite right as well - wonder what would of changed?? I know the client is different in this test. Have to look to getting my between vms speeds back to that over 1Gbps ;)

[Lee Barratt]

My drive pool is setup as a virtual drive. Not Raw mapped. I guess that i cant convert these to raw mapped and will have to take the data off before I can raw map them.

Thanks, will give that a go and get back to you

[Guest]

This is a copy from A Win7 VM to my physical Desktop over a 1GB switch. The discs are thin provisioned.

-------------------------------------------------------------------------------
   ROBOCOPY	 ::	 Robust File Copy for Windows
-------------------------------------------------------------------------------
  Started : 04 January 2013 09:16:13
   Source : VM7COMPUTER
	 Dest : C:\Users\Andrew\Desktop\
	Files : SOME.FILE
  Options : /DCOPY:DA /COPY:DAT /R:1000000 /W:30
------------------------------------------------------------------------------
						   1	VM7COMPUTER

100%		New File			 701.8 m		SOME.FILE
------------------------------------------------------------------------------
			   Total	Copied   Skipped  Mismatch	FAILED	Extras
	Dirs :		 1		 0		 0		 0		 0		 0
   Files :		 1		 1		 0		 0		 0		 0
   Bytes :  701.88 m  701.88 m		 0		 0		 0		 0
   Times :   0:00:06   0:00:06					   0:00:00   0:00:00

   Speed :		   108010209 Bytes/sec.
   Speed :			6180.393 MegaBytes/min.
   Ended : 04 January 2013 09:16:20
[/CODE]

This is copying from a Win XP machine to my physical desktop over the same switch with the VM in the same datastore. Basically, there is nothing different from the above other than its a VM with XP rather than 7.

[CODE]
-------------------------------------------------------------------------------
ROBOCOPY :: Robust File Copy for Windows
-------------------------------------------------------------------------------
Started : 04 January 2013 10:10:43
Source : VMXPCOMPUTER
Dest : C:\Users\Andrew\Desktop\
Files : SOME.FILE
Options : /DCOPY:DA /COPY:DAT /R:1000000 /W:30
------------------------------------------------------------------------------
1 VMXPCOMPUTER
100% New File 851.2 m SOME.FILE
------------------------------------------------------------------------------
Total Copied Skipped Mismatch FAILED Extras
Dirs : 1 0 0 0 0 0
Files : 1 1 0 0 0 0
Bytes : 851.20 m 851.20 m 0 0 0 0
Times : 0:00:13 0:00:13 0:00:00 0:00:00

Speed : 65230532 Bytes/sec.
Speed : 3732.521 MegaBytes/min.
Ended : 04 January 2013 10:10:57
[/CODE]

XP seems to be much slower than its predecessor. I know they are slightly different file sizes but the speed speaks for itself.

TBH as I run with such a small amount of resources I would sooner use XP over 7 yet XP feels much slower than 7 does as a VM on my MicroServer. Windows 2008 R2 feels much snappier than XP does. I need to ditch my XP machine really as these tests above prove 7 is not performing how it should.

The host is a HP MicroServer (N36L so the mk1 version).