190 replies to this topic

[Hoolers]

CPressland, on 16 May 2012 - 10:00, said:

Going through the logs, found this:

2012-05-15T13:45:52.954Z: <<throttled>> Storage I/O Control: connection with vobd failed, error code: -1 errno: 2

Not sure what it is, but it's mentioned CONSTANTLY!

Additionally:

2012-05-16T07:37:08.952Z: [scsiCorrelator] 64332112349us: [vob.scsi.device.io.latency.high] Device t10.ATA_____SAMSUNG_HD502IJ_________________________S13TJDWQ620722______ performance has deteriorated. I/O latency increased from average value of 22658 microseconds to 684718 microseconds.

2012-05-16T07:37:08.952Z: [scsiCorrelator] 64336113058us: [esx.problem.scsi.device.io.latency.high] Device t10.ATA_____SAMSUNG_HD502IJ_________________________S13TJDWQ620722______ performance has deteriorated. I/O latency increased from average value of 22658 microseconds to 684718 microseconds.

2012-05-16T08:03:08.708Z: [scsiCorrelator] 65891868893us: [vob.scsi.device.io.latency.improved] Device t10.ATA_____SAMSUNG_HD502IJ_________________________S13TJDWQ620722______ performance has improved. I/O latency reduced from 684718 microseconds to 174335 microseconds.

More:

My Windows 7 VM is basically completely unresponsive from an I/O standpoint. System is very very slow while copying a VMDK between Datastores. Yet the HDD Indicator light on the Server itself is only flashing once in a while, indicating that minimal I/O is happening on the disk itself.

Sorry mate i am still a bit of a noobie with this stuff but what are you running to see this? I can check and see if my system is doing the same thing? On the ESXi installation i tried that and imported the VM's and saw the same issue.

[+CPressland]

Right so the issue is happening on an ESXi Level, not a VM Level. So further diagnostics on the VMs are pointless.

I've setup a NFS Share on my File Server and attempted to copy over an ISO from one of my DataStores, the speed was averaging about 5% of the 1Gbps connection, 20 minutes for a 3GB file. Everything else can max out this Servers connection easily. I'll attempt to boot a VM over the network later and measure performance again, this will 100% prove is this is a networking issue or an I/O issue as I'll be able to monitor I/O directly on the 2008R2 server.

I'm going to try moving the 'Management Network' to a different LAN and see if that provides me with better performance. Then this weekend I'll install ESXi again on the 250GB Disk that's included with the server and test the same.

[+CPressland]

Hoolers, on 16 May 2012 - 13:02, said:

Sorry mate i am still a bit of a noobie with this stuff but what are you running to see this? I can check and see if my system is doing the same thing? On the ESXi installation i tried that and imported the VM's and saw the same issue.

SSHing into the server, and running cat on the files in /var/log

[+Vice]

Have you considered that maybe there is just an incompatability with your hard drive and ESXi? Have you tried some other type of drive to rule that out? Obviously it is giving you some kind of I/O latency error. What drive is Budman using in his as he seems to have no problems and everything else is identical?

[+CPressland]

My machine has the same drive as Budman's. Additionally, I have two data stores on two separate drives. Both exhibit the same issues. Both are different drive types.

[+Vice]

This is such a weird problem. I don't understand how yours can be not working with identical hardware to budmans. I think honestly that your drives are faulty in some way but I have no way to back that up it's just a gut feeling. I would seriously try some new drives with a fresh installation of ESXi and two copies of Windows. It's time to start ruling stuff out as tweaking settings isn't fixing it.

[Packet1009]

Late entrant to the thread but have been following it on and off.

This may be completely useless, but your mentioning NFS on this page got me thinking - What if you get rid of NFS and use something else? IIRC, the NFS client is v3 for VSphere 5 which means all the traffic for the ESX host uses just 1 single TCP connection.

[Miuku.]

Hoolers, on 10 May 2012 - 10:29, said:

Tried another disk, and deffo a problem with ESX, Same Disk, same hardware with native windows 7 i get about 80 MBPS, with ESXi i get about 10MBPS :-(

ESXi 5, uploading to a datastore via WAN 100Mbps link from home at around 5-6MB/s. Not seeing the issues reported here, DL380G6 servers mostly.

[+BudMan]

Its a given that moving files to the datastore is going to be slow -- this is all over the forums on vmware, etc.

But thats not really the issue we are troubleshooting, the issue comes down to this.

between 2 vms, on the same host, same vswitch using the vmxnet3 drivers -- he is seeing LOW iperf speeds in the range of 300mbps, on the other hand on the exact same hardware N40L

Same test, 2 windows vms on the same host, same vswitch using vmxnet3 drivers I see like 1.+ Gbps

this should have nothing to do with disks, etc.

[+CPressland]

Boom - All non-essential VMs moved to NFS and essential VMs moved to the 500GB Samsung Drive.

I'll zero-fill the SSD and 250GB HDD to ensure disk health and reinstall ESXi onto the 250GB Drive this weekend.

If that fails, I'll get a saucepan, put it on my head, and whack it with a wooden spoon until I no longer care about the performance issues.

[+CPressland]

Guys!!! It's a CPU issue!

Windows 7 Virtual Machine - iperf -s -w 256k

Galactica Physical Machine - iperf -c 10.0.1.11 -w 256k

Speed = Approx 400Mbits/sec

CPU Use = 100% on Windows 7 VM during Transfer.

pfSesne VM - same situation!!!!

So - why is Budman's performance better in this instance? What would cause a VM to use 100% CPU time for a basic network transfer?

[+BudMan]

Do you have your VM's limited to what they can use?

This could be the reason your seeing the limit.

Let me run another iperf and check what my vms cpu does and what it maxes out at, etc.

Ok -- here is client, server and host while doing iperf and getting 1.3+Gbps

client


server


Host


And remember we went over this before, my vms only have 1 cpu assigned to them. and here you can see not restricting their usage on the host



What does your cpu look like -- are you actually hitting 100% and just staying there.. That would explain why your seeing a limit on the speed.

[+CPressland]

Hey,

My VMs only have one core accessible to them. Done a complete reinstall today onto the Physical Hard Drive. So my setup is essentially identical to yours now! Setup the 8GB SSD as a Cache drive.

Can you SSH into your pfSense VM and run top and iperf server on it and let me know what CPU% you get during the transfer. If you get similar issues over pfSense to what I get on Winodws that further helps diagnose this.

Additionally how exactly do you have CPU setup in VM settings?

[+CPressland]

Screenshots:

[+CPressland]

Strange - the Domain Controller VM doesn't get maxed out like the Windows 7 VM does when performing Network Tasks. Did you make any registry tweaks to Networking? I believe you mentioned something about disabling the IPv6 and Tunneling adapters at one point? the Server 2008 R2 is able to get approx 690Mbits/sec compared with 400ish from the Windows 7 VM.

Edit: Server 2003 R2 was able to achieve 772Mbits/sec