esxi ESXi - HP MicroServer N40L Performance

[Vice]

I thought that only allowed you to use it for a 60 day trial until you registered for a license key. :/

[CPressland]

Well, they gave me a licence key, it was free. lol. I was under the impression it's always been. That said, it does have a limited Feature Set.

[Vice]

Who gave you a license key? VMWare? - I asked a friend who has something like 100 servers running this and he says the client software (vSphere Client) has never been free. That if you don't enter a license key (which aren't free) it will stop working after 60 days.

Anyway if you have a valid license I'd suggest contacting VMWare. Even if the hardware isn't supported. I mean you are a paying customer right? You didn't pirate it or anything?

[CPressland]

Yeah, got my Licence from VMWare (not paid), regarding vSphere not being free, I wasn't aware of that. I'd be more than happy to purchase one if I could get this Network Speed issue resolved and they released a Mac Client. If not, CentOS+VirtualBox is the way I shall go.

[Vice]

Earlier in the thread I suggested you install Linux and see if the speed issue remains between Linux guests, did you ever try that?

[Guest]

ESXi can be obtained free. As soon as you add the licence your features are reduced to that of the free licence.

http://www.vmware.com/products/vsphere-hypervisor/overview.html

Product: VMware vSphere 5 Hypervisor Licensed for 1 physical CPUs (unlimited cores per CPU)

License Key: xxxxx-xxxxx-xxxxx-xxxxx-xxxxx

Expires: Never

Product Features:

Up to 32 GB of memory

Up to 8-way virtual SMP

[CPressland]

Earlier in the thread I suggested you install Linux and see if the speed issue remains between Linux guests, did you ever try that?

Two CentOS 6 VMs had the same communication speeds as advertised with Windows VMs.

SK[' timestamp=1336464789' post='594852099]

ESXi can be obtained free. As soon as you add the licence your features are reduced to that of the free licence.

http://www.vmware.co...r/overview.html

Exactly!

[Vice]

Does the free license have any kind of network performance throttling?

[CPressland]

Nope, just 32GB memory limits and 8-way virtual SMP.

[Vice]

Hmmm well this problem sure is interesting. I hope you solve it. :rofl:

[CPressland]

Erm... heres something a bit weird!

I've just downloaded the Ubuntu 12.04 LTS 32-bit ISO and gone to push it to the Server. I attempted to upload to my 250GB Datastore which gave me an ETA of 60 minutes, attempting to go to my 500GB DataStore gave me an ETA of 25 minutes. The 8GB SSD only took 1 minute 30 seconds.

Hmm....

[Hoolers]

Hi,

Firstly thanks for all this :-)

I am getting similar issues with my VMWare install on a N36 and reading this has been a real help in starting to troubleshoot. I have noticed that my server VMDK file on my Samsung SP F4 2TB is performing below par, which manifested itself in very poor copy speeds over the wire (roughly 10MBPS).

I am going to try with another disc before looking at ESX more closely. I'll also try testing the network too although copying from the same hardware on a native windows7 install was much faster so that kind rules out anything but the vswitch.

Look forward to reading updates :-)

Cheers,

Max

[CPressland]

Good to see somebody else is having issues. It looks like it's either an I/O or vSwitch issue at current. I've not yet delved into any advanced troubleshooting or peeling back through logs etc, but naturally I'll update this thread as I know more.

[Hoolers]

Tried another disk, and deffo a problem with ESX, Same Disk, same hardware with native windows 7 i get about 80 MBPS, with ESXi i get about 10MBPS :-(

I have ESXi on a USB key so was gonna do an install on a HDD and see if it makes any difference?

Not happy Jan :-(

[CPressland]

Just a thought - my pfSense is running in promiscuous mode, could this somehow be effecting things?

Could it be that every time I do a network transfer involving these VMs the pfSense VM is receiving every packet pushed over the Virtual Switch?

[+BudMan MVC]

Yeah, if your vswitch is promiscuous mode?? It sure couldn't be helping performance.

Now I just created a new port group to be able to mirror the traffic that goes to the internet for IDS and nchronos, etc. But I haven't installed anything yet to the switch port other than my pfsense lan interface.

So mirrored port group is set to accept promiscuous mode, lan port group is set to reject

But if you have your whole vswitch in that mode, then set promiscuous to reject - does that help? Didn't we go over your vswitch settings already.

[CPressland]

Well - I decided to move some stuff around to see if I could improve performance at all. I moved pfSense to it's own group similar to what you've done and set Promisc to Accept, the rest of the LAN Group is set to Reject.

I'm seeing about 450Mbits/sec over IPerf to most machines on my network on average.

Lets just throw around some other ideas here. I'm starting to think maybe the 8GB SSD is to blame (somehow) at the moment.

If I was to move data to a single Datastore, reinstall ESXi on the provided 250GB HDD and begin setting things up from scratch and reimport the VMs then test performance again?

Or do you think it's worth trying the HP ESXi thats been mentioned multiple times here?

[CPressland]

Extra Note: The Server Completely crashed today when I attempted to download an image from Usenet on my Windows 7 x86 VM. Wasn't responding to any commands, reboot, ssh reboot, etc and required a hard reboot when I got home.

I don't quite see how a VM could bring down the entire Host OS. Either way, I want to explore importing VMs into Xen or VirtualBox and maybe look at moving the VM Platform over to CentOS shortly.

[+BudMan MVC]

Yeah a vm should not be able to bring down the host - that clearly should not happen!! Maybe it was a coincidence? My host crashed last week as well, had to hard reboot the host. Not sure exactly what caused it? I was not even home when it crashed, and systems were not doing any downloads or anything.

As to the HP version of esxi -- a buddy told me it wouldn't install to the N40L.. He recently bought one, but I don't have the details of what he was doing, etc. Could be he was full of **** ;) I have not had time to try it, was thinking I could just install it to a thumb drive to see if worked and added the other hardware to the health tab. But If he was correct I don't think it will -- just that the micro server are not currently supported by HP for esxi it seems. If they were adding the cims should install the needed stuff for the other hardware to be listed.

Doing a clean install to the 250GB drive with 1 datastore would duplicate my install - and sure could see if that bumps your performance up. There is a thread I have been watching on the vmware forums about a guy seeing a limit of ~300Mbps between guests on the same vswitch. Which could be related to your reduced performance - but as of yet nothing of significance in it.

http://communities.vmware.com/thread/392477?start=0&tstart=0

[CPressland]

Yeah a vm should not be able to bring down the host - that clearly should not happen!! Maybe it was a coincidence? My host crashed last week as well, had to hard reboot the host. Not sure exactly what caused it? I was not even home when it crashed, and systems were not doing any downloads or anything.

Well in that case I'll assume it was a simple bug instead of a Load Issue. Going to try and replicate now anyway.

As to the HP version of esxi -- a buddy told me it wouldn't install to the N40L.. He recently bought one, but I don't have the details of what he was doing, etc. Could be he was full of **** ;) I have not had time to try it, was thinking I could just install it to a thumb drive to see if worked and added the other hardware to the health tab. But If he was correct I don't think it will -- just that the micro server are not currently supported by HP for esxi it seems. If they were adding the cims should install the needed stuff for the other hardware to be listed.

Yeah I couldn't see it on the list of Supported Hardware, and Google yielded few results.

Doing a clean install to the 250GB drive with 1 datastore would duplicate my install - and sure could see if that bumps your performance up. There is a thread I have been watching on the vmware forums about a guy seeing a limit of ~300Mbps between guests on the same vswitch. Which could be related to your reduced performance - but as of yet nothing of significance in it.

http://communities.v...tart=0&tstart=0

Right - that's a plan for the weekend then. I'll remove the SSD and attempt an install on the 250GB Drive.

Funny thing is though that the SSD is massively faster than the two hard drives, even when just copying something to the Datastore.

[CPressland]

Do you think it's worth putting pfSense on a different vSwitch all together?

[CPressland]

Hmm - moving the 100GB VMDK that is on my stock 250GB Drive has taken 4 hours thus far. I can move 100GB around on my regular drives in minutes!!! I think the Network and I/O issues are related.

[CPressland]

Going through the logs, found this:

2012-05-15T13:45:52.954Z: <<throttled>> Storage I/O Control: connection with vobd failed, error code: -1 errno: 2[/CODE]

Not sure what it is, but it's mentioned CONSTANTLY!

Additionally:

[CODE]
2012-05-16T07:37:08.952Z: [scsiCorrelator] 64332112349us: [vob.scsi.device.io.latency.high] Device t10.ATA_____SAMSUNG_HD502IJ_________________________S13TJDWQ620722______ performance has deteriorated. I/O latency increased from average value of 22658 microseconds to 684718 microseconds.

2012-05-16T07:37:08.952Z: [scsiCorrelator] 64336113058us: [esx.problem.scsi.device.io.latency.high] Device t10.ATA_____SAMSUNG_HD502IJ_________________________S13TJDWQ620722______ performance has deteriorated. I/O latency increased from average value of 22658 microseconds to 684718 microseconds.

2012-05-16T08:03:08.708Z: [scsiCorrelator] 65891868893us: [vob.scsi.device.io.latency.improved] Device t10.ATA_____SAMSUNG_HD502IJ_________________________S13TJDWQ620722______ performance has improved. I/O latency reduced from 684718 microseconds to 174335 microseconds.
[/CODE]

More:

My Windows 7 VM is basically completely unresponsive from an I/O standpoint. System is very very slow while copying a VMDK between Datastores. Yet the HDD Indicator light on the Server itself is only flashing once in a while, indicating that minimal I/O is happening on the disk itself.

[neoraptor]

at that point, i would start from scratch and check your hdds' health

[CPressland]

at that point, i would start from scratch and check your hdds' health

I know all hard drives are perfectly healthy.

And starting from Scratch won't solve anything at all as this issue has been around since I began working with ESXi, not diagnosing an issue and 'reformatting' is a very bad mentality.