Ramdisk 'root' is full

[Skiver Veteran]

Hi Guys, 

 

I need some help because I seem to be going around in circles and can't quite figure out the root of my issues here. I had this last week but I simply reduced the max file size on a log and it cleared the space and the host could then continue doing what it needed to do.

 

However this time around amending the same file doesn't seem to have solved anything at all.

 

So screenshot heaven time... 

 

First - confirming that the ramdisk is full and the folders under Root and their sizes;

 

The Var folder looks relatively small which then contains logs so that imo doesn't appear to be the issue here but I'll post a screen shot of logs incase I'm missing something;

 

 

 

Comparing this with my host I fixed last week everything looks like they are of similar sizes or smaller (I can post comparison shots if needed). I'm struggling to get my head around the whole scratch setup but see below on that but again I don't think this is anything to worry about;

# esxcli system syslog config get
   Local Log Output: /scratch/log
   Local Logging Default Rotation Size: 1024
   Local Logging Default Rotations: 8
   Log To Unique Subdirectory: false
   Remote Host: <none>

Going back a step, the folder that does seem quite big is proc but that looks like a folder I should not be messing with;

 

 

I'm sure I'm missing something obvious here but I just cannot spot it

[Haggis Veteran]

You do realise that if my manager asks why i have done no work today i am blaming you lol

 

This is confusing me, i have been through everything i know with you and still cant see anything wrong

 

The boot.tgz file and the local.tgz file bother me though i am sure its something with them lol

[n_K]

cd /

du -h -d 1

 

Look which folder has the largest size and investigate there.

/proc is a virtual RAM filesystem, it doesn't really exist, and isn't on your hard drive or storage. Also using ls -l will not get you the size of a folder, they are all the same size folder.

[Haggis Veteran]

I think he has managed to fix it

 

or at least get more info on it

[Skiver Veteran]

Thanks n_K I did discover those commands (not really a Linux nor ESX expert in anyway) so and managed to track down that the file  hostAgentStats-20.stats is the one that seems a little larger than expected.

 

Just need to figure out how to shrink it now

[Haggis Veteran]

Kill it with fire :!

[Skiver Veteran]

Turns out I was wrong, I'm not sure how I got to where I as but now I believe it might be "classSchemas" 

[Torolol]

you probably hits max number of files for that ramdisk implementation.

[Skiver Veteran]

you probably hits max number of files for that ramdisk implementation.

 

Can you expand on that? What files would this happen with? I can edit the amount of logs etc if need be but I'd like to try and fix this intelligently rather than just change stuff till the problem goes away.

 

The biggest Issue I have atm is trying to understand what "folders" are held within the "root ramdisk" For example I can follow the command example above which gives me the below, however quite clearly if my root ramdisk is only 32M in size, it rules out a massive amount of those below. I just feel like I'm going around in circles atm and guessing rather than fully understanding what is held in the ramdisk and how I can intelligently work out what is filling it

~ # du -h -d 1
9.8M    ./etc
497.6M  ./tardisks
4.0K    ./tardisks.noauto
12.0K   ./vmimages
123.7M  ./usr
43.9M   ./sbin
14.2M   ./lib64
12.0K   ./tmp
139.0M  ./lib
4.9M    ./proc
62.5M   ./bin
40.2T   ./dev

[Skiver Veteran]

OK, it seems despite me shrinking the log via Vsphere it doesn't look like it applied it, I manually edited the log file to make it smaller and it seems to have free'd up enough space on the ramdisk.

 

I feel like I'm missing something as this has been a massive pain to figure out - I've always preferred Hyper-V as it seems easier to manage but this seems even more complex than normal to try and resolve. Even though I'd class this as more of a workaround then a fix.

[sc302 Veteran]

IDK, My vsphere servers are pretty much set it and forget it.  I haven't had to go into cli in over a year for any sort of management...then again I have a pretty large raid1 part for the vmware os, and a dedicated datastore partition for the vms.

[Skiver Veteran]

IDK, My vsphere servers are pretty much set it and forget it.  I haven't had to go into cli in over a year for any sort of management...then again I have a pretty large raid1 part for the vmware os, and a dedicated datastore partition for the vms.

 

To be honest most of these hosts have been running for years without needing to even think about this sort of stuff, the Ram disk error is a new one to me. 

 

However it seems to be a fixed size (and pretty small) so it's not really related to the true data store free space, I've not checked that for a few weeks but last time I did check they had a good 5-6 TB free. However these guys did have a habit of going through roughly 1.5TB per month so that could be gone for all I know.

[n_K]

The thing is you're using a ramdisk for /, most ESXi installs wouldn't be using a ramdisk for a root filesystem to store their logs, they'd be using a partition on a local or network disk so that it can be audited and checked, etc.

You get what you pay for, ESXi base is entirely free, hyper-V requires a (quite expensive) windows license for the host server, so you're gonna get a much easier control system for hyper-V, that's not to say that ESXi doesn't have it's benefits, ESXi is a bare-metal hypervisor and so it will always get better performance than a hyper-V server will because it's not driving a graphical GUI or a whole base server OS, and it's got a very powerful GNU/Linux base system which you can do a lot with, but really you need to know what you're doing!

 

One thing that comes to mind with ESXi being strange is copying VMs about, if you do that as you would any normal file using cp, you'll find it's painfully slow - switch to ESXi's built in VM copying tool and you'll see the speed is through the roof. Why is that? Nobody knows, I still can't work out why. But it's quirks like that, which both ESXi and hyper-V will have, that you need to learn to administrate them better .

[sc302 Veteran]

hyper v hypervisor is free as compared to the vmware free hypervisor.  You do not need an expensive windows license to run hyper v

 

https://www.microsoft.com/en-us/evalcenter/evaluate-hyper-v-server-2012-r2

[n_K]

hyper v hypervisor is free as compared to the vmware free hypervisor.  You do not need an expensive windows license to run hyper v

 

https://www.microsoft.com/en-us/evalcenter/evaluate-hyper-v-server-2012-r2

Learn something new each day!

[Skiver Veteran]

The thing is you're using a ramdisk for /, most ESXi installs wouldn't be using a ramdisk for a root filesystem to store their logs, they'd be using a partition on a local or network disk so that it can be audited and checked, etc.

You get what you pay for, ESXi base is entirely free, hyper-V requires a (quite expensive) windows license for the host server, so you're gonna get a much easier control system for hyper-V, that's not to say that ESXi doesn't have it's benefits, ESXi is a bare-metal hypervisor and so it will always get better performance than a hyper-V server will because it's not driving a graphical GUI or a whole base server OS, and it's got a very powerful GNU/Linux base system which you can do a lot with, but really you need to know what you're doing!

 

One thing that comes to mind with ESXi being strange is copying VMs about, if you do that as you would any normal file using cp, you'll find it's painfully slow - switch to ESXi's built in VM copying tool and you'll see the speed is through the roof. Why is that? Nobody knows, I still can't work out why. But it's quirks like that, which both ESXi and hyper-V will have, that you need to learn to administrate them better .

 

This isn't free version I'm afraid, its a pretty expensive one from the little I know about VMware costs as this is tied in with Vcenter and about 17 host with a mix in hardware types but I think the minimum spec has 4 cores. We have a mix (with a favour for Hyper-V) of different environments between Hyper-V and VMware.

 

It is pretty quick but then I wouldn't say our Hyper V system is much slower either with both using similar if not the exact same SANs as datastores.

 

But yeah, I'm learning this stuff as I go, but with our favour on Hyper-V I don't tend to touch VMware too often so when I do have to, added with a real lack of experience in Unix OS's I start to struggle.