Inaccessible VM’s in vSAN

Photo by Jonas Svidras on Unsplash

So in my home lab I had to shutdown my server so I could add some memory and when I started everything back up I noticed that I few of my VM’s in my nested vSAN cluster had inaccessible next to them, so I started Googling around and I found the solution below:

SSH into the vCenter that hosts your vSAN cluster(s) and then execute the below commands:

rvc administrator@vsphere.local@localhost

vsan.check_state -r /localhost/YOUR_DATACENTER/computers/YOUR_VSAN_CLUSTER


Disk not found

So I had an issue trying to upgrade from ESXi 5.5 to 6.0, for some reason the upgrade process would not recognize the RAID 1 that ESXi was installed, after looking around I realized that the HP Smart Array firmware was on 3.x so I loaded the HP SPP which upgraded the firmware to 6.x, and viola, drive showed up during the upgrade process. Lesson here, keep your servers firmware up to date.


Photon OS 1.x /root partition full

Ran into an issue today in which our gitlab server running in a docker container using VMware’s photonOS 1.x ran out of disk space on the /root partition by default this partition is only ~15GB; while the container was still running and I could log into gitlab I couldn’t clone any repos. The solution, simply add space to the disk in vCenter and boot the VM into GParted live an extend /dev/sda1 in my case, reboot, and all was good Thank goodness for GParted Live.


/boot no free space

So I had an issue today in which I kept getting an error message when I attempted to upgrade my ubuntu server, it basically said I was out of disk space, which was odd as I had a 2TB drive as this is a Plex server, after running df -h I quickly noticed that /boot was @ 100% full, so after a quick Google I found the below solution.

If /boot is 100% full you will not be able to just run ‘sudo apt-get autoremove’ you’ll receive a bunch of error messages, so you will have to manually delete some files.

cd /boot

sudo rm abi-4.8.0-53-generic && sudo rm config-4.8.0-53-generic && sudo  rm initrd.img-4.8.0-53-generic && sudo rm System.map-4.8.0-53-generic  &&  sudo rm vmlinuz-4.8.0-53-generic

**The above command must be on 1 line or you can use \ to do multi-line in Linux

Do the above for a few versions(DO NOT DELETE ALL VERSIONS) and then you can run either

sudo apt-get -f install or sudo apt-get autoremove

Also you can run the below commands to see all linux-images and headers on your system

sudo dpkg -l | grep linux-image

sudo dpkg -l | grep linux-headers

dpkg -l linux-image-\* | grep ^ii

And you should now be able to upgrade your system again.


Errors: Invalid operation requested: This ruleset is required and cannot be disabled

So I had to call VMware support today because I noticed on all of our Esxi 6.5 hosts when I attempted to run services.sh restart I received the following message “Errors: Invalid operation requested: This ruleset is required and cannot be disabled” which made no sense to me as I am accustomed to seeing all services restart when I entered that command.  But after a brief communication with VMware support they informed me that that was the new expected behavior for that command, and in order to see the services restart, I need to run the following command ” services.sh restart & tail -f /var/log/jumpstart-stdout.log” and then, of course, you need to hit CTRL+C to exit.  I don’t think I like the new behavior, but at least there’s an easy way to see the services restart.