ESXI VM's inaccessible with black screen after a couple of hours

[+InsaneNutter MVC]

I have a really strange issue with my home ESXi server (ESXi 5.5 2403361), this has run 24/7 for almost three years now without any issues. I've not made any hardware / software changes to it for a long time, or even had to re-boot since I last upgraded to a new ESXi build last January.

 

Essentially the problem I now have is all the VM's running are now becoming inaccessible after a couple of hours. I can't ping any VM's from another device on the network, when i try to view the VM's console from vSphere I just get a black screen and the vSphere client locks up. Likewise I can SSH in to my ESXi server, however if i type anything more than a basic command such as uptime the SSH session will lock up. I can re connect to the ESXi server using a new SSH session / vSphere session, however I cant even re-boot the box as the SSH session just locks up when i try to.

 

It's not like the ESXi server has totally crashed as i can keep re connecting over and over, however I cant really do much other than physically re boot the ESXi server. If I re-boot the server all the VM's will start up ok (Server 2012 R2, Server 2003 and Ubuntu), all these VM's work for a couple of hours, however what I have described above will simply happen again on all the VM's, this is regardless of what OS the VM is running.

 

This first happened a few weeks ago and I simply rebooted the ESXi server, everything was fine for a week. The same then happened again, I rebooted the server and everything was ok for a few days.

 

Now each day when i arrive home from work the same has happened again, it's at the point where it happens every 5-6 hours now.

 

All I can think of is either the SSD the VM's are on is perhaps failing, or the USB drive ESXi is booting from is failing, but if that was the case I cant see why everything would work normally for a couple of hours.

 

Just wondering if anyone has experienced anything like this before?

 

When home tonight I might setup a new USB drive to boot from and start ruling things out, if the same happens again with a new USB boot drive move the VM's off the SSD, then finally test the ram.

 

[sc302 Veteran]

you are on an older build of 5.5, you could try updating the host.

[Jason S. Global Moderator]

The latest 5.5, i believe, is 3024345. That being said, an upgrade might be impossible if it keeps locking up. you have a switch issue?

[sc302 Veteran]

I am on 3116895, if it lasts a day that should be plenty of time to upgrade.

[n_K]

Go to the server and go to the live error log (think it might be ctrl + f8? Try ctrl+f1-f12) and see if there are any errors. Don't think the SSD would be failing, you could yank it out mid-operating and the VMs would be screwed but ESXi should still function and be connectable...

As far as I'm aware, ESXi loads most of itself into RAM on bootup so it shouldn't be using your USB much after that unless you're doing exotic things

 

My first suspicion would be RAM, then USB, then another hardware issue.

[+BudMan MVC]

So you have vserver running so u dont have free version cant u just call vmware for support?

[+InsaneNutter MVC]

Thanks for the input, I have upgraded to 3116895. I guess time will tell if that's changed anything, if not ill leave memtest running tomorrow night and go from there.

 

Not sure what you mean Budman? My ESXi box is called "virtualserver", I have the standard free licence you can get.

[+BudMan MVC]

So u use vclient to connect u dont use vcenter to manage

 

once u get the hangs worked out u should update to 6 which is paste update 1 already

[+InsaneNutter MVC]

Are you on a mobile Budman? it's unlike you to use abbreviations  such as "u".

 

The two network interfaces on the servers motherboard are Realtek r8168's, these had drivers out the box on ESXi 5.0 and 5.1, however the drivers were removed from ESXi 5.5.

 

With ESXi 6.0 the drivers are blacklisted, apparently a work around exist but its not something I have looked in to yet.  With that in mind and the fact users of free licences are limited to the vSphere Client, which cant configure any of the newer features I never really felt the need to update to 6.0.

 

Maybe things have now changed for free users? I must be honest i've not really been following ESXi news for a while. ESXi 5.5 has always been good enough for my home usage scenario at the moment.

[binaryzero]

If you've got NIC problems, instead of trying to hack at it I'd just buy a new NIC. HCL all the way  

[+InsaneNutter MVC]

Indeed I agree with hindsight only using hardware on the HCL would have been the better choice, everything did work out the box with the standard ESXi ISO on 5.0 / 5.1 and I could upgrade to 5.5 without loosing the existing network drivers. The motherboard is a Mini-ITX and the 1x PCIe port is been used for the raid controller, so no chance of changing anything there. I don't think network drivers are the issue though as ESXI 5.5 has worked perfectly since release until now, and I can still access ESXi over the network when the VM's are not responding.

 

My VM's became inaccessible again last night unfortunately.

 

I'm thinking it could be the SSD as when viewing the logs on screen this morning I was seeing seeing "below MEDIA WEAROUT threshold (0)" for the SSD, although it's strange I cant power up less important VM's which are not on the SSD.

 

I didn't have much time to do much before work this morning, however i've left memtest running which will hopefully rule out any memory problems when i'm home tonight.

[+InsaneNutter MVC]

Just to update - memtest found no problems with the ram, so i moved the VM's on the SSD across to a different datastore. Since doing that ESXi has been up for around 18 hours with no issues so far (fingers crossed).

 

Also for anyone reading this using a free VMware licence, VmWare Labs has released a free tool which now allows to manage ESXi host via web interface (using HTML 5 and javascript) without the need of vCenter: Free ESXi web interface.

[binaryzero]

^ that's cool, home labbers will be happy.

 

Vcenter all the way though, haha. 

[+BudMan MVC]

yeah Im am traveling this week for work.  sorry about the U, its a bad habit when texting.. and using my phone keyboard.

 

Yeah that is a fling that was put out https://labs.vmware.com/flings/esxi-embedded-host-client

 

While agreed some of the stuff in 6 is meant for enterprise solutions..  And you don't have access to these using vclient, etc..  There are plenty of other things that make it worth being on current version.  NVMe for example NFS 4.1 as another..

 

I like to stay current..  You can still manage your esxi with vclient,  And sure you could use the fling to not even need the vclient..  As to the limits to hardware 8 stuff vs 11, the stuff is there and available you can just not easy edit with the client..  But there are other ways to get that stuff going, its not like it restricted you just can not use the vclient to point and click to set that stuff up,.