esxi ESXi - HP MicroServer N40L Performance

[Hoolers]

Going through the logs, found this:

2012-05-15T13:45:52.954Z: <<throttled>> Storage I/O Control: connection with vobd failed, error code: -1 errno: 2[/CODE]

Not sure what it is, but it's mentioned CONSTANTLY!

Additionally:

[CODE]
2012-05-16T07:37:08.952Z: [scsiCorrelator] 64332112349us: [vob.scsi.device.io.latency.high] Device t10.ATA_____SAMSUNG_HD502IJ_________________________S13TJDWQ620722______ performance has deteriorated. I/O latency increased from average value of 22658 microseconds to 684718 microseconds.

2012-05-16T07:37:08.952Z: [scsiCorrelator] 64336113058us: [esx.problem.scsi.device.io.latency.high] Device t10.ATA_____SAMSUNG_HD502IJ_________________________S13TJDWQ620722______ performance has deteriorated. I/O latency increased from average value of 22658 microseconds to 684718 microseconds.

2012-05-16T08:03:08.708Z: [scsiCorrelator] 65891868893us: [vob.scsi.device.io.latency.improved] Device t10.ATA_____SAMSUNG_HD502IJ_________________________S13TJDWQ620722______ performance has improved. I/O latency reduced from 684718 microseconds to 174335 microseconds.
[/CODE]

More:

My Windows 7 VM is basically completely unresponsive from an I/O standpoint. System is very very slow while copying a VMDK between Datastores. Yet the HDD Indicator light on the Server itself is only flashing once in a while, indicating that minimal I/O is happening on the disk itself.

Sorry mate i am still a bit of a noobie with this stuff but what are you running to see this? I can check and see if my system is doing the same thing? On the ESXi installation i tried that and imported the VM's and saw the same issue.

[CPressland]

Right so the issue is happening on an ESXi Level, not a VM Level. So further diagnostics on the VMs are pointless.

I've setup a NFS Share on my File Server and attempted to copy over an ISO from one of my DataStores, the speed was averaging about 5% of the 1Gbps connection, 20 minutes for a 3GB file. Everything else can max out this Servers connection easily. I'll attempt to boot a VM over the network later and measure performance again, this will 100% prove is this is a networking issue or an I/O issue as I'll be able to monitor I/O directly on the 2008R2 server.

I'm going to try moving the 'Management Network' to a different LAN and see if that provides me with better performance. Then this weekend I'll install ESXi again on the 250GB Disk that's included with the server and test the same.

[CPressland]

Sorry mate i am still a bit of a noobie with this stuff but what are you running to see this? I can check and see if my system is doing the same thing? On the ESXi installation i tried that and imported the VM's and saw the same issue.

SSHing into the server, and running cat on the files in /var/log

[Vice]

Have you considered that maybe there is just an incompatability with your hard drive and ESXi? Have you tried some other type of drive to rule that out? Obviously it is giving you some kind of I/O latency error. What drive is Budman using in his as he seems to have no problems and everything else is identical?

[CPressland]

My machine has the same drive as Budman's. Additionally, I have two data stores on two separate drives. Both exhibit the same issues. Both are different drive types.

[Vice]

This is such a weird problem. I don't understand how yours can be not working with identical hardware to budmans. I think honestly that your drives are faulty in some way but I have no way to back that up it's just a gut feeling. I would seriously try some new drives with a fresh installation of ESXi and two copies of Windows. It's time to start ruling stuff out as tweaking settings isn't fixing it.

[Packet1009]

Late entrant to the thread but have been following it on and off.

This may be completely useless, but your mentioning NFS on this page got me thinking - What if you get rid of NFS and use something else? IIRC, the NFS client is v3 for VSphere 5 which means all the traffic for the ESX host uses just 1 single TCP connection.

[Miuku.]

Tried another disk, and deffo a problem with ESX, Same Disk, same hardware with native windows 7 i get about 80 MBPS, with ESXi i get about 10MBPS :-(

ESXi 5, uploading to a datastore via WAN 100Mbps link from home at around 5-6MB/s. Not seeing the issues reported here, DL380G6 servers mostly.

[+BudMan MVC]

Its a given that moving files to the datastore is going to be slow -- this is all over the forums on vmware, etc.

But thats not really the issue we are troubleshooting, the issue comes down to this.

between 2 vms, on the same host, same vswitch using the vmxnet3 drivers -- he is seeing LOW iperf speeds in the range of 300mbps, on the other hand on the exact same hardware N40L

Same test, 2 windows vms on the same host, same vswitch using vmxnet3 drivers I see like 1.+ Gbps

this should have nothing to do with disks, etc.

[CPressland]

Boom - All non-essential VMs moved to NFS and essential VMs moved to the 500GB Samsung Drive.

I'll zero-fill the SSD and 250GB HDD to ensure disk health and reinstall ESXi onto the 250GB Drive this weekend.

If that fails, I'll get a saucepan, put it on my head, and whack it with a wooden spoon until I no longer care about the performance issues.

[CPressland]

Guys!!! It's a CPU issue!

Windows 7 Virtual Machine - iperf -s -w 256k

Galactica Physical Machine - iperf -c 10.0.1.11 -w 256k

Speed = Approx 400Mbits/sec

CPU Use = 100% on Windows 7 VM during Transfer.

pfSesne VM - same situation!!!!

So - why is Budman's performance better in this instance? What would cause a VM to use 100% CPU time for a basic network transfer?

[+BudMan MVC]

Do you have your VM's limited to what they can use?

This could be the reason your seeing the limit.

Let me run another iperf and check what my vms cpu does and what it maxes out at, etc.

Ok -- here is client, server and host while doing iperf and getting 1.3+Gbps

client

server

Host

And remember we went over this before, my vms only have 1 cpu assigned to them. and here you can see not restricting their usage on the host

What does your cpu look like -- are you actually hitting 100% and just staying there.. That would explain why your seeing a limit on the speed.

[CPressland]

Hey,

My VMs only have one core accessible to them. Done a complete reinstall today onto the Physical Hard Drive. So my setup is essentially identical to yours now! Setup the 8GB SSD as a Cache drive.

Can you SSH into your pfSense VM and run top and iperf server on it and let me know what CPU% you get during the transfer. If you get similar issues over pfSense to what I get on Winodws that further helps diagnose this.

Additionally how exactly do you have CPU setup in VM settings?

[CPressland]

Screenshots:

[CPressland]

Strange - the Domain Controller VM doesn't get maxed out like the Windows 7 VM does when performing Network Tasks. Did you make any registry tweaks to Networking? I believe you mentioned something about disabling the IPv6 and Tunneling adapters at one point? the Server 2008 R2 is able to get approx 690Mbits/sec compared with 400ish from the Windows 7 VM.

Edit: Server 2003 R2 was able to achieve 772Mbits/sec

[+BudMan MVC]

here is my win7 client to pfsense running iperf server

But I have never seen great performance to and from windows with pfsense, but as you see my windows client is only see about 50% cpu and 500Mbps -- which is much better than yours????

My pfsense is using the e1000 drivers because I was having an issue with access to outside vpn from my laptop when pfsense used the vmxnet3 drivers.

Have not had time to dig into why.. But I think it might be related to the way pfsense labels the interfaces while using this drivers vmx3f0 and vmx3f1 -- which looks to cause a problem with the vlan stuff, and might have be related??? All I know for sure is an ipsec\L2TP vpn to an outside server from a client on my network doesn't work with the vmxnet3 drivers.

Maybe I have tweaked my windows install different than you? I was playing with adapter settings the other day and netsh global options, RSS, Chimney offload, etc I was able to get 1.7Gbps the other day while playing. Now only seeing like 1.3 -- I might of reset something or made another tweak without check performance?

Your limits seem real close to that vmware thread I saw about the guy seeing a 300Mbps limit on his vswitches.

Maybe we should use a specific linux live cd or something as our benchmark for vswitch performance.. Could just fire up 2 exact vms that way and compare performance.

I believe I am running a different version of pfsense than you for one thing. I am running 2.1 32bit, with latest gitsync - its running 8.3 freebsd, etc.

[+BudMan MVC]

networking settings are different in 2k8 vs w7 for sure -- even just out of the gate.

But yeah I have disabled all the ipv6 stuff on my windows vms. But I do run it on my workstation, and my linux vm and pfsense. But sure its been correctly setup ;) ie the teredo and 6to4 **** disabled because not using that!

as I mentioned in previous post - maybe picking a linux live distro to boot would be a better apples to apples test.

[CPressland]

YEAAAAAAHHHHH!!!

Enabled VMCI and Communication between VMs just went through the roof!

That combined with the Server 2003 R2 general networking performance means this is somehow now only effecting the Windows 7 VM. And that I can more than live with.

So, for the VMs dealing with downloading I'm going to allocate 2 CPUs so one can handle the Downloading and Networking while the other can handle FileSystem and File Operations (PAR+RAR).

Now I've just gotta figure out how to get Server 2003 R2 to actually see the two CPUs without doing a full reinstall.

[CPressland]

as I mentioned in previous post - maybe picking a linux live distro to boot would be a better apples to apples test.

Sure - Arch might be the best bet because we can turn basically everything off on a tiny install. Want me to build a VMDK and send you the package?

[+BudMan MVC]

so you enabled vmci -- let me try that ;)

[CPressland]

so you enabled vmci -- let me try that ;)

Forces VM comms to skip the networking stack from what I hear. Solved my issues for the most part anyway :)

[+BudMan MVC]

nice

C:\Windows\System32>iperf -c storage.local.lan -w 256k

------------------------------------------------------------

Client connecting to storage.local.lan, TCP port 5001

TCP window size: 256 KByte

------------------------------------------------------------

[268] local 192.168.1.216 port 49182 connected with 192.168.1.8 port 5001

[ ID] Interval Transfer Bandwidth

[268] 0.0-10.0 sec 1.95 GBytes 1.67 Gbits/sec

Now I need to put my tweaks back - I had backed them all out, atleast the ones I could remember and was seeing like 1.15Gbps -- now its freaking rockin!! Lets see what happens when I put some of the tweaks back.. Lets see if can get that 1.8 or 2 mark ;)

[CPressland]

Make notes so I can do the same ;)

[+BudMan MVC]

ok check this out -- still playing with settings

C:\Windows\System32>iperf -c storage.local.lan -w 256k

------------------------------------------------------------

Client connecting to storage.local.lan, TCP port 5001

TCP window size: 256 KByte

------------------------------------------------------------

[268] local 192.168.1.216 port 49307 connected with 192.168.1.8 port 5001

[ ID] Interval Transfer Bandwidth

[268] 0.0-10.0 sec 2.17 GBytes 1.86 Gbits/sec

But seems disable interrupt moderation on the virtual nic sure doesn't hurt ;)

[CPressland]

Nice speeds there! Make sure to post a list of Tweaks!