ESXI VM's inaccessible with black screen after a couple of hours


Recommended Posts

I have a really strange issue with my home ESXi server (ESXi 5.5 2403361), this has run 24/7 for almost three years now without any issues. I've not made any hardware / software changes to it for a long time, or even had to re-boot since I last upgraded to a new ESXi build last January.

 

Essentially the problem I now have is all the VM's running are now becoming inaccessible after a couple of hours. I can't ping any VM's from another device on the network, when i try to view the VM's console from vSphere I just get a black screen and the vSphere client locks up. Likewise I can SSH in to my ESXi server, however if i type anything more than a basic command such as uptime the SSH session will lock up. I can re connect to the ESXi server using a new SSH session / vSphere session, however I cant even re-boot the box as the SSH session just locks up when i try to.

 

It's not like the ESXi server has totally crashed as i can keep re connecting over and over, however I cant really do much other than physically re boot the ESXi server. If I re-boot the server all the VM's will start up ok (Server 2012 R2, Server 2003 and Ubuntu), all these VM's work for a couple of hours, however what I have described above will simply happen again on all the VM's, this is regardless of what OS the VM is running.

 

This first happened a few weeks ago and I simply rebooted the ESXi server, everything was fine for a week. The same then happened again, I rebooted the server and everything was ok for a few days.

 

Now each day when i arrive home from work the same has happened again, it's at the point where it happens every 5-6 hours now.

 

All I can think of is either the SSD the VM's are on is perhaps failing, or the USB drive ESXi is booting from is failing, but if that was the case I cant see why everything would work normally for a couple of hours.

 

Just wondering if anyone has experienced anything like this before?

 

When home tonight I might setup a new USB drive to boot from and start ruling things out, if the same happens again with a new USB boot drive move the VM's off the SSD, then finally test the ram.

 

2rqgc5t.jpg

Go to the server and go to the live error log (think it might be ctrl + f8? Try ctrl+f1-f12) and see if there are any errors. Don't think the SSD would be failing, you could yank it out mid-operating and the VMs would be screwed but ESXi should still function and be connectable...

As far as I'm aware, ESXi loads most of itself into RAM on bootup so it shouldn't be using your USB much after that unless you're doing exotic things

 

My first suspicion would be RAM, then USB, then another hardware issue.

Thanks for the input, I have upgraded to 3116895. I guess time will tell if that's changed anything, if not ill leave memtest running tomorrow night and go from there.

 

Not sure what you mean Budman? My ESXi box is called "virtualserver", I have the standard free licence you can get.

Are you on a mobile Budman? it's unlike you to use abbreviations  such as "u".

 

The two network interfaces on the servers motherboard are Realtek r8168's, these had drivers out the box on ESXi 5.0 and 5.1, however the drivers were removed from ESXi 5.5.

 

With ESXi 6.0 the drivers are blacklisted, apparently a work around exist but its not something I have looked in to yet.  With that in mind and the fact users of free licences are limited to the vSphere Client, which cant configure any of the newer features I never really felt the need to update to 6.0.

 

Maybe things have now changed for free users? I must be honest i've not really been following ESXi news for a while. ESXi 5.5 has always been good enough for my home usage scenario at the moment.

Indeed I agree with hindsight only using hardware on the HCL would have been the better choice, everything did work out the box with the standard ESXi ISO on 5.0 / 5.1 and I could upgrade to 5.5 without loosing the existing network drivers. The motherboard is a Mini-ITX and the 1x PCIe port is been used for the raid controller, so no chance of changing anything there. I don't think network drivers are the issue though as ESXI 5.5 has worked perfectly since release until now, and I can still access ESXi over the network when the VM's are not responding.

 

My VM's became inaccessible again last night unfortunately.

 

I'm thinking it could be the SSD as when viewing the logs on screen this morning I was seeing seeing "below MEDIA WEAROUT threshold (0)" for the SSD, although it's strange I cant power up less important VM's which are not on the SSD.

 

I didn't have much time to do much before work this morning, however i've left memtest running which will hopefully rule out any memory problems when i'm home tonight.

Just to update - memtest found no problems with the ram, so i moved the VM's on the SSD across to a different datastore. Since doing that ESXi has been up for around 18 hours with no issues so far (fingers crossed).

 

Also for anyone reading this using a free VMware licence, VmWare Labs has released a free tool which now allows to manage ESXi host via web interface (using HTML 5 and javascript) without the need of vCenter: Free ESXi web interface.

yeah Im am traveling this week for work.  sorry about the U, its a bad habit when texting.. and using my phone keyboard.

 

Yeah that is a fling that was put out https://labs.vmware.com/flings/esxi-embedded-host-client

 

While agreed some of the stuff in 6 is meant for enterprise solutions..  And you don't have access to these using vclient, etc..  There are plenty of other things that make it worth being on current version.  NVMe for example NFS 4.1 as another..

 

I like to stay current..  You can still manage your esxi with vclient,  And sure you could use the fling to not even need the vclient..  As to the limits to hardware 8 stuff vs 11, the stuff is there and available you can just not easy edit with the client..  But there are other ways to get that stuff going, its not like it restricted you just can not use the vclient to point and click to set that stuff up,.

This topic is now closed to further replies.
  • Recently Browsing   0 members

    • No registered users viewing this page.
  • Posts

    • OpenAI is now rolling out Lockdown Mode to more ChatGPT users by Pradeep Viswanathan Back in February, OpenAI first announced Lockdown Mode for users who want the most comprehensive protection from potential attacks. At the time of the announcement, the company mentioned that this feature was available to a small set of highly security-conscious users, including executives or security teams at leading organizations. Today, OpenAI announced that Lockdown Mode is now rolling out to all personal ChatGPT accounts, including Free, Go, Plus, and Pro, and also self-serve ChatGPT Business accounts. Users can enable the feature from ChatGPT Settings > Security when it is available for their account. When Lockdown Mode is enabled, ChatGPT limits or disables several features that connect to the web or external services. These include live web browsing, Deep Research, Agent Mode, and more. Here is the complete list of services that will be disabled in Lockdown Mode: Live web browsing: Web browsing is limited to accessing only cached content. Search results may be limited, unavailable, or stale. Image support: ChatGPT may not display images in regular responses or retrieve images from the web. Users can still upload image files, and image generation remains available where it is otherwise available. Deep research: Deep research is disabled. Agent mode: Agent mode is disabled. Canvas networking: Users cannot approve Canvas-generated code to access the network. File downloads: ChatGPT cannot download files for data analysis. ChatGPT can still operate on your manually uploaded files. It is important to note that Lockdown Mode does not completely block prompt injections from appearing in content that ChatGPT processes. For example, a malicious instruction could still be present in an uploaded file or cached web content. However, the mode is designed to reduce the ways such an attack could send sensitive information outside the conversation. Along with Lockdown Mode, OpenAI today also announced that the Active sessions feature is now available across ChatGPT accounts and workspace types. This feature allows users to review where their account is signed in across devices and end sessions if required. A session will have the following information displayed: Device or browser information. First-party app context, such as ChatGPT, Codex, or API Platform. Approximate location. Sign-in date and time. Whether the device is a trusted device. Whether it is your current session. OpenAI highlighted that the Active sessions feature will not be available for accounts linked to an organization’s single sign-on setup, including SAML or OIDC.
    • with LSTC and ESU, moving to w11 or linux because w10 suddenly will not work when in reality it works and its a better choice, of course there are also developers that only test in 11 or force you to have TPM and Secure boot for the sake of "better security" in games. or most likely people is buying new PC that only ship with 11
    • with LTSC and ESU there are still viable as a stable platform not that they care and let people deal with w11 crashing and burning every month support mean shet if the platform is trash
    • Most boring game ever. Repetitive, empty, predictable, and full of cliches. Total waste of time and money, IMO.
    • Mafia: The Old Country expansion Man of Honor announced, brings back Salieri from original by Pulasthi Ariyasinghe During Summer Game Fest, 2K and Hanger 13 brought out a new Mafia: The Old Country trailer, revealing the game's first expansion. Named Man of Honor, this is slated to add two new chapters to the Enzo storyline that the game follows. There is an iconic character returning to the series with this expansion, with players set to run into Ennio Salieri, the future Don of the Salieri crime family. Fans of the original Mafia, or its Definitive Edition remake, may remember that name as one of the biggest characters in the storyline. This expansion is set prior to his rise to being the kingpin in the City of Lost Heaven. "Set in Sicily during the winter of 1905, Enzo Favara has proven himself a reliable soldier of the Torrisi crime family in the months since his initiation," says the studio about the new chapters. "Now, the Don entrusts him and Cesare with a delicate assignment of assisting Ennio Salieri, a man of honor recently released from prison and intent on reclaiming what is his." Working at Salieri's side, players will be heading into fresh environments as they return to the role of Enzo as a high-ranking soldato. The studio also promises brand-new weapons, fresh vehicles, and charms to collect in this expansion. Moreover, the expansion will add new content to the updated Free Ride mode. Alongside new collectibles and locations, this will add more challenges to beat alongside Salieri, which are described as runs that will "test the skills of even the most elite mafiosi." The Mafia: The Old Country Man of Honor story expansion will release on August 14, 2026, across PC, Xbox Series X|S, and PlayStation 5. It will cost $10 for owners of the base game to jump into.
  • Recent Achievements

    • Rookie
      moog19 went up a rank
      Rookie
    • Mentor
      grik went up a rank
      Mentor
    • Dedicated
      JKR earned a badge
      Dedicated
    • One Year In
      CHUNWEI earned a badge
      One Year In
    • Conversation Starter
      FBSPL earned a badge
      Conversation Starter
  • Popular Contributors

    1. 1
      +primortal
      488
    2. 2
      PsYcHoKiLLa
      270
    3. 3
      Skyfrog
      75
    4. 4
      Steven P.
      68
    5. 5
      FloatingFatMan
      63
  • Tell a friend

    Love Neowin? Tell a friend!