ESXI VM's inaccessible with black screen after a couple of hours


Recommended Posts

I have a really strange issue with my home ESXi server (ESXi 5.5 2403361), this has run 24/7 for almost three years now without any issues. I've not made any hardware / software changes to it for a long time, or even had to re-boot since I last upgraded to a new ESXi build last January.

 

Essentially the problem I now have is all the VM's running are now becoming inaccessible after a couple of hours. I can't ping any VM's from another device on the network, when i try to view the VM's console from vSphere I just get a black screen and the vSphere client locks up. Likewise I can SSH in to my ESXi server, however if i type anything more than a basic command such as uptime the SSH session will lock up. I can re connect to the ESXi server using a new SSH session / vSphere session, however I cant even re-boot the box as the SSH session just locks up when i try to.

 

It's not like the ESXi server has totally crashed as i can keep re connecting over and over, however I cant really do much other than physically re boot the ESXi server. If I re-boot the server all the VM's will start up ok (Server 2012 R2, Server 2003 and Ubuntu), all these VM's work for a couple of hours, however what I have described above will simply happen again on all the VM's, this is regardless of what OS the VM is running.

 

This first happened a few weeks ago and I simply rebooted the ESXi server, everything was fine for a week. The same then happened again, I rebooted the server and everything was ok for a few days.

 

Now each day when i arrive home from work the same has happened again, it's at the point where it happens every 5-6 hours now.

 

All I can think of is either the SSD the VM's are on is perhaps failing, or the USB drive ESXi is booting from is failing, but if that was the case I cant see why everything would work normally for a couple of hours.

 

Just wondering if anyone has experienced anything like this before?

 

When home tonight I might setup a new USB drive to boot from and start ruling things out, if the same happens again with a new USB boot drive move the VM's off the SSD, then finally test the ram.

 

2rqgc5t.jpg

Go to the server and go to the live error log (think it might be ctrl + f8? Try ctrl+f1-f12) and see if there are any errors. Don't think the SSD would be failing, you could yank it out mid-operating and the VMs would be screwed but ESXi should still function and be connectable...

As far as I'm aware, ESXi loads most of itself into RAM on bootup so it shouldn't be using your USB much after that unless you're doing exotic things

 

My first suspicion would be RAM, then USB, then another hardware issue.

Thanks for the input, I have upgraded to 3116895. I guess time will tell if that's changed anything, if not ill leave memtest running tomorrow night and go from there.

 

Not sure what you mean Budman? My ESXi box is called "virtualserver", I have the standard free licence you can get.

Are you on a mobile Budman? it's unlike you to use abbreviations  such as "u".

 

The two network interfaces on the servers motherboard are Realtek r8168's, these had drivers out the box on ESXi 5.0 and 5.1, however the drivers were removed from ESXi 5.5.

 

With ESXi 6.0 the drivers are blacklisted, apparently a work around exist but its not something I have looked in to yet.  With that in mind and the fact users of free licences are limited to the vSphere Client, which cant configure any of the newer features I never really felt the need to update to 6.0.

 

Maybe things have now changed for free users? I must be honest i've not really been following ESXi news for a while. ESXi 5.5 has always been good enough for my home usage scenario at the moment.

Indeed I agree with hindsight only using hardware on the HCL would have been the better choice, everything did work out the box with the standard ESXi ISO on 5.0 / 5.1 and I could upgrade to 5.5 without loosing the existing network drivers. The motherboard is a Mini-ITX and the 1x PCIe port is been used for the raid controller, so no chance of changing anything there. I don't think network drivers are the issue though as ESXI 5.5 has worked perfectly since release until now, and I can still access ESXi over the network when the VM's are not responding.

 

My VM's became inaccessible again last night unfortunately.

 

I'm thinking it could be the SSD as when viewing the logs on screen this morning I was seeing seeing "below MEDIA WEAROUT threshold (0)" for the SSD, although it's strange I cant power up less important VM's which are not on the SSD.

 

I didn't have much time to do much before work this morning, however i've left memtest running which will hopefully rule out any memory problems when i'm home tonight.

Just to update - memtest found no problems with the ram, so i moved the VM's on the SSD across to a different datastore. Since doing that ESXi has been up for around 18 hours with no issues so far (fingers crossed).

 

Also for anyone reading this using a free VMware licence, VmWare Labs has released a free tool which now allows to manage ESXi host via web interface (using HTML 5 and javascript) without the need of vCenter: Free ESXi web interface.

yeah Im am traveling this week for work.  sorry about the U, its a bad habit when texting.. and using my phone keyboard.

 

Yeah that is a fling that was put out https://labs.vmware.com/flings/esxi-embedded-host-client

 

While agreed some of the stuff in 6 is meant for enterprise solutions..  And you don't have access to these using vclient, etc..  There are plenty of other things that make it worth being on current version.  NVMe for example NFS 4.1 as another..

 

I like to stay current..  You can still manage your esxi with vclient,  And sure you could use the fling to not even need the vclient..  As to the limits to hardware 8 stuff vs 11, the stuff is there and available you can just not easy edit with the client..  But there are other ways to get that stuff going, its not like it restricted you just can not use the vclient to point and click to set that stuff up,.

This topic is now closed to further replies.
  • Recently Browsing   0 members

    • No registered users viewing this page.
  • Posts

    • Alphabet Raises $84.75 Billion To Feed Its AI Compute Hunger https://www.techtimes.com/articles/317953/20260607/alphabet-raises-8475-billion-feed-its-ai-compute-hunger-warren-buffetts-berkshire-buying.htm
    • Good point Anti-spoofing protection for cloud mailboxes Anti-spoofing technology in Microsoft 365 specifically examines forgery of the From header field (also known as the 5322.From address, From address, or P2 sender), because email clients show the From header value as the message sender. When Microsoft 365 has high confidence the From header is forged, the message is identified as spoofed. https://learn.microsoft.com/en...g-protection-spoofing-about
    • I sort of agree with you on that. I had a telescope - a real hefty thing, although only around 500CHF - that got me so fascinated about the stars and planets. I would stare for hours, amazed that I could see the craters of the moon or the rings of Saturn in "realtime" (quotations because, y'know, speed of light and whatnot). A friend of mine has a telescope like the one mentioned in the review, and the pictures are amazing to look at. But there is something missing for me. I may as well just go to NASA's website and look through their gallery.
    • As opposed to catching bad press because it's the engine's fault? The engine has all the tools a dev needs to optimize their game, Epic isn't gonna hold their hand to make sure they use them. Also, Epic isn't forcing anyone to use Lumen, Nanite or super high resolution meshes and textures. Just because Nanite can render meshes with millions of polygons doesn't mean you should use it like that, in fact, Epic recommends you avoid doing that. Most of the stuttering can easily be fixed by cutting down on unique materials by using master materials, using instanced meshes, generating a PSO cache, pre-compiling every shader before loading the game and also managing what must be loaded and unloaded at runtime. It's up to the developer to set all that up properly.
    • I think it depends on what you're looking for to do, and the time you have to spare. With my Dwarf 3, I easily spend 3-4 hour sessions; half an hour driving to an un-light polluted place, another half hour unpacking and setting up the smart scope + tripod for equatorial tracking, then 15 more minutes mucking around with settings and shooting calibration frames, spending a few hours shooting, merging with past photo sessions, etc. It's crazy how time flies and I often get home later than I expected. It's something I still need to set aside a good part of an evening to do, all in all. For one session, where you often need like four for best results when it comes to deep space objects. Even with a smart scope like Dwarf 3, regular non-astro photography is still way more approachable to people getting into photography. I find this is a time consuming niche no matter how I go about it. With practice, I can probably begin cutting time here but I think where smart scopes find their home is among people who love to shoot the night sky but don't have the spare time to go deep with the "navigator level" attunement to the night sky itself in addition to everything else. Having said this, _if_ you have even more time to spend on this hobby, it will probably be even more rewarding to do it more by hand and learn the skies and the details of how it all works.
  • Recent Achievements

    • One Month Later
      DJC50PLUS earned a badge
      One Month Later
    • Week One Done
      DJC50PLUS earned a badge
      Week One Done
    • Proficient
      Eric Biran went up a rank
      Proficient
    • Dedicated
      Conjor earned a badge
      Dedicated
    • Week One Done
      Windows Guy earned a badge
      Week One Done
  • Popular Contributors

    1. 1
      +primortal
      493
    2. 2
      PsYcHoKiLLa
      246
    3. 3
      Steven P.
      72
    4. 4
      +Edouard
      69
    5. 5
      neufuse
      68
  • Tell a friend

    Love Neowin? Tell a friend!