VMware

VM ran out of disk space, but not the guest OS

I’m not sure what happened here, and I’m hoping reddit can help me figure it out.

* Centos 7 VM, 128GB disk, thin provisioned. ESXi 6.5 only (no Vcenter)
* ESXi stopped the VM, and displayed the question “the vm is out of space, would you like to try the last operation again, or cancel”.
* I ssh’ed into the guest OS (hitting retry about 100 times) and deleted some files. No change. Guest OS said it was using 27 GB of 128GB.
* ESXi said there was 60GB of free space on the datastore.
* There were no snapshots of the VM

I was able to fix the issue by exporting the VM, then redeploying it (again thin provisioned). I just don’t know what the original issue was. I still have the old VM if I need to power it on again.

Any thoughts as to what cause this issue?


View Reddit by HotButteredGopherView Source

Related Articles

7 Comments

  1. Your issue is the thin provisioning. VMware will present 128GB to the OS even if there is not 128GB worth of space left on the datastore. Once the datastore fills up…game over, you need more physical storage. This is why thin provisioning is both a blessing and a curse. I work for an MSP and there is no way we can keep track of storage like that for 300+ clients so by rule of thumb we ALWAYS use thick provisioning. This way the OS will get it’s space and it will be “earmarked” on the datastore, if it’s not available on the datastore ESXi won’t let you create the disk.

    Also, remember that ESXi stores a “page file” for your memory usage on your VM. I’m assuming you aren’t using 60+ GB RAM for Cent OS but who knows.

  2. We had this issue several times in one of our environments. Thin provisioned .vmdk on a datastore with 5TB free complaining about not enough room too grow the disk. Support confirmed it was a bug in VMFS6. We were running 6.5u2 at the time. They said it was supposed to be resolved in 6.5u3 or 6.7u2.

  3. Deleting files on the guest OS does not reduce the space allocated on the storage managed by ESXi

    So you’re going to need more space on the storage side or delete the VM and create a smaller one.

  4. Sounds like a bug. Check the release notes for patch levels upwards from your current or search the VMware kb using Google and the error you received to figure out which update fixes it.

  5. Did somebody recently grow the datastore it is hosted on? I have seen issues in 6 where the datastore is grown and vcenter reports the new size. However individual hosts are unable to see the free space.

    You might get an error like this:

    Error Stack: The disk extend operation failed: 14 (Bad address)
    Task Details:
    Name: reconfigure
    Status: Insufficient disk space on datastore ”.

    To fix it you can remove the VM from inventory. Perform individual storage rescan on each host in clusterand then re-add the VM back to inventory.

    You can then grow the disk and monitor the vmware.log

    **2019-08-05T14:58:07.305Z| vcpu-0| I125: DISKLIB-LIB : Growing disk ‘/vmfs/volumes/5c0ff2fb-fe023933-ccda-8cdcd4b7b8f8/USUATDB02/USUATDB02_1.vmdk’ : createType = vmfs**
    **2019-08-05T14:58:07.305Z| vcpu-0| I125: DISKLIB-LIB : Growing disk ‘/vmfs/volumes/5c0ff2fb-fe023933-ccda-8cdcd4b7b8f8/USUATDB02/USUATDB02_1.vmdk’ : capacity = 629145600 sectors – 300 GB**
    **2019-08-05T14:58:07.305Z| vcpu-0| I125: DISKLIB-LIB : Growing disk ‘/vmfs/volumes/5c0ff2fb-fe023933-ccda-8cdcd4b7b8f8/USUATDB02/USUATDB02_1.vmdk’ : new capacity = 1153433600 sectors – 550 GB**

Leave a Reply

Your email address will not be published. Required fields are marked *

Close