esxi ESXi - HP MicroServer N40L Performance

[Guest]

I have the N36L version of this box. Love it to bits for playing about on. Running ESXi 5 and yes the slowest thing in this box really are the drives. I have three discs and try to stagger my VM's across them so no disc is really doing too much work.

I've have had 1 x 2008 R2 DC, 2 x Exchange 2010 running on 2008 R2, 1 x Windows 7 and a Arch Linux. They run well as long as each one isnt under too much load (Disc I/O).

Anyone managed to get OSX working? Works fine on a machine in my office running an Intel CPU but on this Microserver it just sits with a grey apple logo. I figure its the AMD CPU and pretty much given up trying now.

[+John Teacake MVC]

Here is the ram I ordered -- cheaper, and have had not issues with it.

20-148-347 MEM 4Gx2 | CRUC CT2KIT51264BA1339 R $42.99

http://www.newegg.co...N82E16820148347

Looks like the price went up a couple of bucks.

Thanks! I can install vSphere 5 then now. Got it from the UK site however!

[remixedcat]

If anyone needs discounts on any vmware stuff I can help since I am a partner.

[CPressland]

I just have to serve up the files via smb.. I don't have to transcode them on the fly or anything.. Just a simple file server, my popcorn hour plays the files in whatever format they are in.

So you were writing 3 different installs to the same datastore at the same time - and reading all from the same iso?? WTF yeah that going to be slow as hell! Why not just install once and then copy the install if you were installing the same OS?

Yeah, I used to have a Popcorn Hour but didn't like the UI. Will be migrating everything over to Raspberry Pi with XBMC/Plex shortly anyway. That said, I've not used a Popcorn Hour since 2008. I'm guessing they've improved somewhat since then? All I remember about it was power cycling it about 50 times a day.

I would think the NL40 is quite limited in regards to performance, it would only take a number of VM's for it to choke.

Well, the only VM thats really doing any work is the pfSense VM and the Sabnzbd/Sick Beard VM, running 6 total.

"it would only take a number of VM's for it to choke."

No **** ;) But you can say that for any hardware - the number of VMs you can run at the same time is just going to vary. And what those VM's are doing.

What I can dell you is it runs my router, and is running my file server, and my test box w7 and my linux distro. The file server, the router and the linux box are on all the time..

It is more than capable of handling the number of VMs I would ever need to be running at the same time, and allow for many many more to be available when I need them for play or testing. Allowed me to combine my file server and my router boxes into a tiny little thing - that looking currently at the killawatt meter its connected to is drawing 55.5 watts ;) That is with 4 disks in it. Its been on for 366 hours and has drawn a total of 20.3KWH, so I pay like 12.5 cents per kwh so your looking at $2.50 to operate for 15 days. So about $5 a month to have it on running all the machines I need to run or play with.

And for the price you sure and the hell can not beat it for a home esxi solution or even a small business setup. No I wouldn't expect to run my enterprise VMs on it ;)

Looks like I found a pretty big part of what was happening with the I/O problems too.

See the Disk I/O. Notice the drop, thats because one of my Hard Drives had some damaged sectors and ESXi wasn't marking them as bad on the fly, causing massive I/O and Error Rates, after yanking the drive it dropped massively and all VMs are running great now.

SK[' timestamp=1335380702' post='594824025]

I have the N36L version of this box.

I have a N36L as my primary storage server, Windows Server 2008 R2, Software RAID5 across 4 Drives (2TB). 5.5TB Total Storage.

SK[' timestamp=1335380702' post='594824025]

Anyone managed to get OSX working? Works fine on a machine in my office running an Intel CPU but on this Microserver it just sits with a grey apple logo. I figure its the AMD CPU and pretty much given up trying now.

OS X runs fine with ESXi on a XServe / Mac Pro. But it doesn't even want to attempt a startup on this HP Hardware. Would require a hacked Kernel I think, and I wouldn't touch that with a bargepole!

If anyone needs discounts on any vmware stuff I can help since I am a partner.

Nice offer! But ESXi is Free. That said I will likely need some Workstation Licences in the coming months so I may touch base with you on that at some point.

[CPressland]

Just as a further update to this thread.

My VSwitch is configured to run at Full Gigabit, MTU Set at 9000. But I am unable to get more than 100Mbit out of the VMs, see below.

ideas?

As you can also see, the individual NICs are set to Gigabit speeds also.

[CPressland]

Hmm, changed the adapter from E1000 to VMXNET3 Same thing, 10MB/s maximum.

Oh wow! 13MB/s!

But on my Physical Machine (Mac Mini, I can pull 75MB/s easily)

[Guest]

I get the same problem, to be honest I haven't looked into it yet, would be interested if you find a fix. it was doing it over iSCSI from a Windows 7 to FreeNAS VM for me.

[+BudMan MVC]

Why do you have your mtu at 9000? Do you have jumbo frames setup on your nics? Does you physical switch support jumbo?

BTW - what is producing that stats page? Showing your disk io and network and memory?

Your graph is terrible why is it all over the place? See mine that is from the VM, and that copy dialog is from my workstation. Also notice my VM nic is showing connected at 10GB, not just 1 ;) I never changed the mtu from default 1500 -- there is NO reason too. Does every device on your network have a 9000 mtu? If not your switch is doing a **** load of work.

[n_K]

From a VM transferring a file to the host ESXi server via SCP I get about 2MB/s... I'm not even kidding, it's abismal.

From one VM host to another on a private internal 'network' I get about 18MB/s.

That's on a dual quare core 8GB server with 2 VMs

[Guest]

I know when I tried to set my box to 9000 it said the nic didnt support it.

[CPressland]

SK[' timestamp=1335438001' post='594825411]

I get the same problem, to be honest I haven't looked into it yet, would be interested if you find a fix. it was doing it over iSCSI from a Windows 7 to FreeNAS VM for me.

Keep an eye on this thread then.

Why do you have your mtu at 9000? Do you have jumbo frames setup on your nics? Does you physical switch support jumbo?

BTW - what is producing that stats page? Showing your disk io and network and memory?

Your graph is terrible why is it all over the place? See mine that is from the VM, and that copy dialog is from my workstation. Also notice my VM nic is showing connected at 10GB, not just 1 ;) I never changed the mtu from default 1500 -- there is NO reason too. Does every device on your network have a 9000 mtu? If not your switch is doing a **** load of work.

I changed up my MTU to 9000 to see if that helped increase the speed, and it did, by 3MB/s. So I've lowered it back down.

How on earth did you get a 10Gbit to show up in ESXi?

Regarding the Graph. Thats VMWares Client for the iPad, the graphs were all over the place until I ripped out that dodgy hard drive, now they're holding stable with no issues.

[CPressland]

From a VM transferring a file to the host ESXi server via SCP I get about 2MB/s... I'm not even kidding, it's abismal.

Yeah, transferring directly to ESXi gives me horrid performance too, took about 3 hours to transfer my Server 2008 r2 image over the Console, and about an hour and a half using SFTP.

SK[' timestamp=1335438801' post='594825437]

I know when I tried to set my box to 9000 it said the nic didnt support it.

My entire network supports 9000, I just don't use it normally, this was the one case where I did.

[+BudMan MVC]

10GB is autoneg with driver for vmxnet3

here

yeah putting files on the datastore directly blows chunks.. I found a lot of threads about ftp server on the older 4.1 esxi that helped, but have not found good thread about putting ftp on esxi 5 yet.

But really only files you have to put there are iso for your installs and patches/upgrades to esxi -- so not too worried about slow speed uploading files to the datastore - not something have to do very often.

[Guest]

VMware told me that you should always use the vmxnet3 card if you can as there are actually performance benefits. Typically only supported on the newer Windows systems though.

[CPressland]

Oh Snap, mines 10Gbit now, must have happened when I shifted the driver to vmxnet3.

Anyway, that doesn't solve the issue. Any other ideas? Shall I post up some ESXi Networking Config?

[+BudMan MVC]

It would be supported by anything that you can install the tools too - this has the drivers for it. I had them running on my pfsense box (freebsd 8.3) until I ran into a weird issue with connecting to ipsec/l2tp vpn from behind my router.

Freaking strangest thing -- I have to spend some time documenting it so can put in a bug report. But I could just not get a connection, it would just hang - I even turn on my oakley.log in windows to see the details of the ipsec connection, and nothing looked wrong - it just would hang. As soon as I put the nics back to e1000 on my pfsense vm - bam connected right into my work vpn from my phsycial laptop.

Let me grab a large file from vmfs, my transfers are all too and from RDM drives.. I assume yours are from a vmfs disk?

[CPressland]

Freaking strangest thing -- I have to spend some time documenting it so can put in a bug report. But I could just not get a connection, it would just hang - I even turn on my oakley.log in windows to see the details of the ipsec connection, and nothing looked wrong - it just would hang. As soon as I put the nics back to e1000 on my pfsense vm - bam connected right into my work vpn from my phsycial laptop.

Yeah I decided against using the vmxnet3 drivers on FreeBSD for the time being.

[+BudMan MVC]

Ok that is pretty bad! This is writing to my vmfs ? of my vm..

Now I run my system drives on my vms as thin provisioned.. So I am quite sure its having to expand out the file system as I copy that file. Let me do a copy back after it gets up there, and I can test read speed from it and then writing file when the file system has already been expanded out, etc.

Ok this copy file from vmfs -- not as good as the RDM drive, and it took a while to ramp up see the networking graph. But its workable I guess. But RDM drive is much better ;) Maybe that is your issue. Give the vm raw access to the disk vs a vmfs.

Well thats not too shaby there.. I deleted that 3gb file I had copied up to the vmfs, so it would of expanded the drive. Then copied up a different iso (backtrack) and 70MBps -- can live with that for sure.

[CPressland]

Well, thats pretty terrible, and the same kinda speeds I've been seeing.

The Server 2003 R2 box I've setup for sabnzbd was choking, but as the file system has been expanded it's got better.

The Problem comes in as

Incomplete Downloads go to a eager zeroed VMFS

Once downloaded they expand on this VMFS

then move to a Downloads_Complete folder located on a N36L with a dedicated Windows Server 2008 R2 OS

Finally, Sickbeard picks up the file and moves this to a Media Folder after doing some processing on the files.

Annoyingly, I can download a 300MB file faster than I can move it across my network in some instances.

[Guest]

*Edit*

Ignore me. Think were better with thick discs then? :/

[+BudMan MVC]

see my edit.. I just wrote a file back to my vmfs after it had expanded with the 3GB file copy up. Now I see 70MB+ writing to that file system.

[Guest]

Agh, still, a nice find though!

Thin works for me since the discs I have are the default one and two I found laying around :p

2 x 150GB

1 x 75GB

I should invest in some new discs but I already spent my quota this year on a new gaming PC.

[CPressland]

Maybe worth running a filesystem zeroing utility on each VM then just to expand them out to full size.

That doesn't explain my issues though.

Recommend any Free-Space ZeroFilling Tools?

[CPressland]

http://technet.microsoft.com/en-us/sysinternals/bb897443 <-- this seams to work well.

[+BudMan MVC]

Ok I did copy back from that vmfs, and its rocking.. I even rebooted my workstation to make sure nothing was cached since I had just copied up that iso of backtrack, etc..

Can you do a simple iperf test to your vm.. is the bottle neck network or disk??

So as you can see, I see between 750 and 800Mbps with iperf to my vm from my workstation. What do you get?

Just simple math of 750/8 tells me I should at max there be able to copy at 93MBps -- i am seeing 70 to 80.. So works out pretty well. And when I use that nastester I do believe its close to 90 even.