makeThin Documentation by Ruben Miguelez Garcia is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License
While it is relatively easy to convert the vDisks of your VMs from thick to thin using SVMotion (Storage VMotion), this method is not always applicable and cannot be used in an automated batch process.
makeThin was created to address this limitation in LabManager environments but can be used in others as well.
makeThin allows you to automatically discover thick vDisks and convert them to thin in a batch, supervised process. The process is fully verifiable and can be reverted if it is needed.
This document describes the makeThin program and other functions associated together with its usage, reason of existence and some results.
1. Disclaimer
This program and its documentation is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
2. Assumptions
I assume that you are used to working on a Linux shell and that you have previous knowledge about what the VM files are for.
I also assume you know what the difference between a thin provisioned and thick vDisk is
3. Conventions
Throughout this document I will use some short terms to describe different things:
- Convert:
-
Act of applying the
makeThinscript to one or more vDisks. - Clean up:
-
Act of running the
cleanupSAFETMPafter a successful conversion. - vDisk
- Virtual disk, usually attached to a VM (Virtual Machine).
4. Reason to create this program
The reason was efficiency in the use of storage and LabManager. LabManager has no problem using configurations (groups of VMs in a can, similar to vApp) with vms that have thin vDisks, but when you import a configuration from one LabManager into another LabManager, all the vms will be created using thick disks, independly of the original configuration. When you escalate this to dozens of LabManager configurations, each one with 3 to 7 vms across several LabManager environments, the waste of GBs is considerable.
Since there is no performance penalty on using thin disks in a LabManager environment, then there is no reason not to use them across all datastores.
So, when first I came across this challenge I considered using SVMotion, but I found that it didn't provide an automated solution since some manual steps where required to leave everything as LabManager expects. Also because LabManager structure of folders with numbers makes really difficult to know where your VMs are located in case you only want to convert a few of them.
So I started digging and found a very simple way to convert, in a batch process, as many vDisks as you wanted, with no errors, no hassle and very few requirements.
My team and I have been using this program for almost a year and so far we haven't experienced any issue. In fact, 'thinning the configuration' has become almost a part of the procedure when importing configurations.
The savings in GBs depend on how you create & use your VMs, but in my case they are huge (>50%).
5. Requirements
5.1. ESX requirements
This program has been tested on VMware ESX 4.x but should work without problems on ESX 3.5
ESXi has a reduced shell and makeThin as it is will not work there.
Remember that from one host you can convert all the vDisks on all the datastores visible from that host.
5.2. VMs requirements
The vDisks that you convert, must not be in use while you convert them. In other words, the VM/VMs that use those vDisks must be powered off. A very easy way to verify this is to navigate to vSphere Client > Datastores > Virtual Machines tab. If you see something running there, then you know what to do.
 |
5.3. Datastore requirements
This may not be an absolute truth, but using this script in principle only makes sense if your datastores are on VMFS and not NFS. This is because in NFS the way the blocks of a file are allocated on the filesystem is decided by the NFS server and not by the ESX.
I haven't tried to use this on NFS, but you can test and tell me the results.
6. Installation
Really simple:
- Copy the source code into the filesystem of your ESX.
You can do this in many different ways:
- Run
# curl -O location_of_file_on_webif your Service Console has connection to Internet; OR - Download the source code to your desktop and using vSphere client browse a datastore and copy it there; OR
- Use WinSCP to copy the source code from your desktop to the ESX.
- Run
- Load into the shell the functions the source code contains with with the "." command:
# . /path/to/source_code
Example (for those that don't see the period before the file name)
# . makeThin_source_code_v6.txt
There is no output after running this command.
Now you can start using the functions from the SSH console where you run the command on step 2.
Note that the functions are loaded on that specific shell while it is alive, if you open another shell you will have to repeat step 2.
7. How to use makeThin and its related programs
Below I will explain how to use makeThin. Because makeThin does not actually 'convert' the vDisk but instead create a thin clone that replaces the original vDisk, after a successful conversion you need to clean up the original thick vDisks, for that we will use cleanupSAFETMP. Once you are done with the converting and cleaning, you can use computeThin to evaluate the savings you have achieved.
From this point on, I assume that you have successfully installed the program and loaded its functions/scripts into the shell.
7.1. Using makeThin
Through this documentation I use the makeThin easy name but the original name in the source code is findAndMakeDiskThin. You can use both names.
makeThin findAndMakeDiskThin # Usage: findAndMakeDiskThin /path/ # It will find Base Disks recursively under the path specified and one by one, convert them to thin provision ONLY if required and ONLY if you confirm the operation. # To say YES you just need to press y or Y, without Enter. # Any other key, including Enter will mean NO.
The /path/ parameter accepts normal bash wildcards, just make sure you know what you are doing.
It will do some checks, display some info and ask for confirmation, for every disk. This is because I have no way to verify if a file is locked, so visual analysis and confirmation is needed.
Lets see an example.
# cd /vmfs/volumes/vCR2/euclid-vcr/
# findAndMakeDiskThin 44[5-9]
446/000446-import-0-0-scsi.vmdk is thinProvisioned, skipping.
-----------------------------------------------------------------------------
Working with 447/000447-import-0-0-scsi.vmdk. Maximum space needed: 40G
Filesystem Size Used Avail Use% Mounted on
/vmfs/volumes/4c52ebfc-90b8db3c-bf58-001f295a6e9c/euclid-vcr/447
1.2T 920G 279G 76% /vmfs/volumes/vCR2
Verify the file is not in use
447/000447-import-0-0-scsi-flat.vmdk: x86 boot sector, Microsoft Windows XP MBR, Serial 0xc440c440; partition 1: ID=0x7, active, starthead 1, startsector 63, 83875302 sectors
-- Convert to thin? (y/n): y
Will run: vmkfstools -E "447/000447-import-0-0-scsi.vmdk" "447/SAFETMP000447-import-0-0-scsi.vmdk"
Will run: vmkfstools -i "447/SAFETMP000447-import-0-0-scsi.vmdk" -d thin "447/000447-import-0-0-scsi.vmdk"
Destination disk format: VMFS thin-provisioned
Cloning disk '447/SAFETMP000447-import-0-0-scsi.vmdk'...
Clone: 100% done.
Visual verification
-rw------- 1 root root 42949672960 Nov 28 12:25 447/000447-import-0-0-scsi-flat.
-rw------- 1 root root 542 Nov 28 12:35 447/000447-import-0-0-scsi.vmdk
-rw------- 1 root root 42949672960 Aug 4 12:56 447/SAFETMP000447-import-0-0-scs
-rw------- 1 root root 523 Nov 28 12:25 447/SAFETMP000447-import-0-0-scs
-----------------------------------------------------------------------------
[...]
As you can see, it displays some information about the Datastore where we are, and the size of the vDisk we want to convert. If you are very short of space, it may be a better idea to start with small vDisks, so you don't run out of space. The maximum size a thin vDisk can reach is the size of the thick vDisk, that's what the "Maximum space needed" means.
The program does the hard work for you, but you still have to think what you are doing and plan properly. Another script that we will see later (computeThin) can help you with your planning.
How does a locked/not locked file look like?
On ESX 4.x
- Locked
# file VC400-flat.vmdk VC400-flat.vmdk: writable, regular file, no read permission
- Not locked
# file vDisk-flat.vmdk vDisk-flat.vmdk: data OR # file 002767-import-0-0-scsi-flat.vmdk 002767-import-0-0-scsi-flat.vmdk: x86 boot sector, Microsoft Windows XP MBR, Serial 0xc440c440; partition 1: ID=0x7, active, starthead 1,
However, and this is very important, if the VM has snapshots, the snapshot(s) the vmx is pointing to will be locked, but the BaseDisk(s) will NOT be locked. So once again, make sure the VM is down. If you use linked clones, then any VM that uses that BaseDisk must be down.
7.2. Using cleanupSAFETMP
Because makeThin does not convert the vDisk in place but rather does a clone and the script does not delete anything, you need to clean up the original thick files after proper verification.
cleanupSAFETMP # Usage: cleanupSAFETMP /path/ # Find SAFETMP*vmdk files recursively under the path specified and ask you for confirmation before deleting them.
If no path is provided, then it will search from ".", which is the path where you are when you run the command. Here is the example
# cleanupSAFETMP
Files found:
./448/SAFETMP000448-import-0-0-scsi-flat.vmdk
./448/SAFETMP000448-import-0-0-scsi.vmdk
./446/SAFETMP000446-import-0-1-scsi-flat.vmdk
./446/SAFETMP000446-import-0-1-scsi.vmdk
./447/SAFETMP000447-import-0-0-scsi-flat.vmdk
./447/SAFETMP000447-import-0-0-scsi.vmdk
Do you want to delete these files?
Answer (y/n): y
Done.
Anything different from y/Y means NO, even Enter alone means NO.
This deletion of files cannot be reverted.
7.3. Using computeThin
I wanted to give some figures about how much space this script was saving.
Firstly I though on using the output of # vdf -h before and after, but then I thought the command that gives apparent and real size of a file (du) would be a much precise solution.
computeThin # Usage: computeThin /path/ # Using the linux command 'du' it will calculate the difference between real size and apparent size in terms of blocks used. # The comparison is valid once the file is thin provision, otherwise both results are the same.
Examples
# pwd /vmfs/volumes/vCR3 # computeThin . Thin Thick File 9.5G 15G ./euclid-vcr/3379/003379-vCenter_4.1-flat.vmdk 1.3G 8.0G ./euclid-vcr/3379/003379-vCenter_4.1_1-flat.vmdk 4.2G 20G ./euclid-vcr/3378/003378-import-0-0-scsi-flat.vmdk 8.1G 20G ./euclid-vcr/3377/003377-import-0-0-scsi-flat.vmdk 7.0G 15G ./euclid-vcr/3376/003376-ESXi_41-2-0-scsi-flat.vmdk 864M 15G ./euclid-vcr/3376/003376-import-0-0-scsi-flat.vmdk 3.3G 10G ./euclid-vcr/3376/003376-ESXi_41-1-0-scsi-flat.vmdk 3.8G 15G ./euclid-vcr/3184/003184-import-0-0-scsi-flat.vmdk 4.8G 12G ./euclid-vcr/3185/003185-import-0-0-scsi-flat.vmdk 1.0G 1.0G ./euclid-vcr/3185/003185-import-0-1-scsi-flat.vmdk 18G 40G ./euclid-vcr/3186/003186-import-0-0-scsi-flat.vmdk 18G 40G ./euclid-vcr/3187/003187-import-0-0-scsi-flat.vmdk 3.8G 20G ./euclid-vcr/3188/003188-import-0-0-scsi-flat.vmdk 3.7G 15G ./euclid-vcr/3189/003189-import-0-0-scsi-flat.vmdk 4.8G 12G ./euclid-vcr/3190/003190-import-0-0-scsi-flat.vmdk 1.0G 1.0G ./euclid-vcr/3190/003190-import-0-1-scsi-flat.vmdk 18G 40G ./euclid-vcr/3191/003191-import-0-0-scsi-flat.vmdk 3.7G 15G ./euclid-vcr/3192/003192-import-0-0-scsi-flat.vmdk 4.8G 12G ./euclid-vcr/3193/003193-import-0-0-scsi-flat.vmdk 1.0G 1.0G ./euclid-vcr/3193/003193-import-0-1-scsi-flat.vmdk 18G 40G ./euclid-vcr/3194/003194-import-0-0-scsi-flat.vmdk 18G 40G ./euclid-vcr/3195/003195-import-0-0-scsi-flat.vmdk 3.8G 20G ./euclid-vcr/3196/003196-import-0-0-scsi-flat.vmdk 18G 40G ./euclid-vcr/3197/003197-import-0-0-scsi-flat.vmdk 3.8G 20G ./euclid-vcr/3198/003198-import-0-0-scsi-flat.vmdk 6.5G 15G ./euclid-vcr/3202/003202-import-0-0-scsi-flat.vmdk 4.8G 12G ./euclid-vcr/3203/003203-import-0-0-scsi-flat.vmdk 1.0G 1.0G ./euclid-vcr/3203/003203-import-0-1-scsi-flat.vmdk 18G 40G ./euclid-vcr/3204/003204-import-0-0-scsi-flat.vmdk 18G 40G ./euclid-vcr/3205/003205-import-0-0-scsi-flat.vmdk 371M 15G ./euclid-vcr/3206/003206-import-0-0-scsi-flat.vmdk 4.8G 12G ./euclid-vcr/3207/003207-import-0-0-scsi-flat.vmdk 1.0G 1.0G ./euclid-vcr/3207/003207-import-0-1-scsi-flat.vmdk 3.5G 15G ./euclid-vcr/3208/003208-import-0-0-scsi-flat.vmdk 18G 40G ./euclid-vcr/3209/003209-import-0-0-scsi-flat.vmdk 4.8G 12G ./euclid-vcr/3210/003210-import-0-0-scsi-flat.vmdk 1.0G 1.0G ./euclid-vcr/3210/003210-import-0-1-scsi-flat.vmdk 18G 40G ./euclid-vcr/3211/003211-import-0-0-scsi-flat.vmdk 18G 40G ./euclid-vcr/3212/003212-import-0-0-scsi-flat.vmdk 18G 40G ./euclid-vcr/3213/003213-import-0-0-scsi-flat.vmdk 3.4G 20G ./euclid-vcr/3214/003214-import-0-0-scsi-flat.vmdk 971M 30G ./euclid-vcr/3267/003267-W2003_SP2_64bit-0-0-scsi-flat.vmdk 6.1G 12G ./euclid-vcr/3375/003375-import-0-0-scsi-flat.vmdk 1.0G 1.0G ./euclid-vcr/3375/003375-import-0-1-scsi-flat.vmdk 3.6G 15G ./euclid-vcr/4714/004714-import-0-0-scsi-flat.vmdk 4.8G 12G ./euclid-vcr/4715/004715-import-0-0-scsi-flat.vmdk 1.0G 1.0G ./euclid-vcr/4715/004715-import-0-1-scsi-flat.vmdk 18G 40G ./euclid-vcr/4716/004716-import-0-0-scsi-flat.vmdk 18G 40G ./euclid-vcr/4717/004717-import-0-0-scsi-flat.vmdk 4.9G 20G ./euclid-vcr/4718/004718-import-0-0-scsi-flat.vmdk 0 23G ./euclid-vcr/4719/004719-import-0-1-scsi-flat.vmdk 14G 40G ./euclid-vcr/4719/004719-import-0-0-scsi-flat.vmdk 14G 40G ./euclid-vcr/4749/004749-import-0-0-scsi-flat.vmdk 0 23G ./euclid-vcr/4749/004749-import-0-1-scsi-flat.vmdk 3.5G 20G ./euclid-vcr/4748/004748-import-0-0-scsi-flat.vmdk 18G 40G ./euclid-vcr/4747/004747-import-0-0-scsi-flat.vmdk 18G 40G ./euclid-vcr/4746/004746-import-0-0-scsi-flat.vmdk 4.8G 12G ./euclid-vcr/4745/004745-import-0-0-scsi-flat.vmdk 1.0G 1.0G ./euclid-vcr/4745/004745-import-0-1-scsi-flat.vmdk 379M 15G ./euclid-vcr/4737/004737-import-0-0-scsi-flat.vmdk 1.3G 8.0G ./windows_server_2003_standard_x64/windows_server_2003_standard_x64_1-flat.vmdk 9.7G 15G ./windows_server_2003_standard_x64/windows_server_2003_standard_x64-flat.vmdk Thin: 462G total Thick: 1.3T total
# cd ../vCR2/ # computeThin . Thin Thick File 7.7G 40G ./euclid-vcr/194/000194-import-0-0-scsi-flat.vmdk 0 23G ./euclid-vcr/194/000194-import-0-1-scsi-flat.vmdk 3.4G 20G ./euclid-vcr/193/000193-import-0-0-scsi-flat.vmdk 18G 40G ./euclid-vcr/192/000192-import-0-0-scsi-flat.vmdk 18G 40G ./euclid-vcr/191/000191-import-0-0-scsi-flat.vmdk 4.8G 12G ./euclid-vcr/190/000190-import-0-0-scsi-flat.vmdk 1.0G 1.0G ./euclid-vcr/190/000190-import-0-1-scsi-flat.vmdk 361M 15G ./euclid-vcr/189/000189-import-0-0-scsi-flat.vmdk 18G 40G ./euclid-vcr/532/000532-import-0-0-scsi-flat.vmdk 4.8G 12G ./euclid-vcr/531/000531-import-0-0-scsi-flat.vmdk 1.0G 1.0G ./euclid-vcr/531/000531-import-0-1-scsi-flat.vmdk 18G 40G ./euclid-vcr/448/000448-import-0-0-scsi-flat.vmdk 361M 15G ./euclid-vcr/445/000445-import-0-0-scsi-flat.vmdk 4.8G 12G ./euclid-vcr/446/000446-import-0-0-scsi-flat.vmdk 1.0G 1.0G ./euclid-vcr/446/000446-import-0-1-scsi-flat.vmdk 18G 40G ./euclid-vcr/447/000447-import-0-0-scsi-flat.vmdk 13G 40G ./euclid-vcr/535/000535-import-0-0-scsi-flat.vmdk 0 23G ./euclid-vcr/535/000535-import-0-1-scsi-flat.vmdk 3.4G 20G ./euclid-vcr/534/000534-import-0-0-scsi-flat.vmdk 18G 40G ./euclid-vcr/533/000533-import-0-0-scsi-flat.vmdk 361M 15G ./euclid-vcr/530/000530-import-0-0-scsi-flat.vmdk 3.5G 20G ./euclid-vcr/449/000449-import-0-0-scsi-flat.vmdk 3.5G 20G ./euclid-vcr/536/000536-import-0-0-scsi-flat.vmdk 3.0G 20G ./euclid-vcr/455/000455-import-0-0-scsi-flat.vmdk 18G 40G ./euclid-vcr/454/000454-import-0-0-scsi-flat.vmdk 18G 40G ./euclid-vcr/453/000453-import-0-0-scsi-flat.vmdk 4.8G 12G ./euclid-vcr/452/000452-import-0-0-scsi-flat.vmdk 1.0G 1.0G ./euclid-vcr/452/000452-import-0-1-scsi-flat.vmdk 361M 15G ./euclid-vcr/451/000451-import-0-0-scsi-flat.vmdk 14G 40G ./euclid-vcr/450/000450-import-0-0-scsi-flat.vmdk 0 23G ./euclid-vcr/450/000450-import-0-1-scsi-flat.vmdk 18G 40G ./euclid-vcr/3155/003155-import-0-0-scsi-flat.vmdk 4.8G 12G ./euclid-vcr/3199/003199-import-0-0-scsi-flat.vmdk 1.0G 1.0G ./euclid-vcr/3199/003199-import-0-1-scsi-flat.vmdk 18G 40G ./euclid-vcr/3156/003156-import-0-0-scsi-flat.vmdk 7.0G 15G ./euclid-vcr/3153/003153-import-0-0-scsi-flat.vmdk 18G 40G ./euclid-vcr/3201/003201-import-0-0-scsi-flat.vmdk 18G 40G ./euclid-vcr/2766/002766-import-0-0-scsi-flat.vmdk 18G 40G ./euclid-vcr/2767/002767-import-0-0-scsi-flat.vmdk 4.8G 12G ./euclid-vcr/2765/002765-import-0-0-scsi-flat.vmdk 1.0G 1.0G ./euclid-vcr/2765/002765-import-0-1-scsi-flat.vmdk 7.4G 15G ./euclid-vcr/2764/002764-2329-ESXi-flat.vmdk 4.8G 12G ./euclid-vcr/3154/003154-import-0-0-scsi-flat.vmdk 1.0G 1.0G ./euclid-vcr/3154/003154-import-0-1-scsi-flat.vmdk 18G 40G ./euclid-vcr/3200/003200-import-0-0-scsi-flat.vmdk 5.1G 20G ./euclid-vcr/3157/003157-import-0-0-scsi-flat.vmdk 1.5G 4.0G ./t_XPsp3sm/t_XPsp3sm-flat.vmdk 2.8G 8.0G ./t_win2k3sp2sm/t_win2k3sp2sm-flat.vmdk Thin: 367G total Thick: 1.1T total
# cd ../vCR # computeThin . Thin Thick File 103M 103M ./euclid-vcr/24/000024-VMwareLM-ServiceVM-I17-H10-0-0-scsi-flat.vmdk -> I didn't want to touch these. 103M 103M ./euclid-vcr/23/000023-VMwareLM-ServiceVM-I17-H13-0-0-scsi-flat.vmdk 103M 103M ./euclid-vcr/22/000022-VMwareLM-ServiceVM-I17-H16-0-0-scsi-flat.vmdk 103M 103M ./euclid-vcr/21/000021-VMwareLM-ServiceVM-I17-H15-0-0-scsi-flat.vmdk 103M 103M ./euclid-vcr/20/000020-VMwareLM-ServiceVM-I17-H12-0-0-scsi-flat.vmdk 103M 103M ./euclid-vcr/19/000019-VMwareLM-ServiceVM-I17-H9-0-0-scsi-flat.vmdk 103M 103M ./euclid-vcr/18/000018-VMwareLM-ServiceVM-I17-H14-0-0-scsi-flat.vmdk 103M 103M ./euclid-vcr/17/000017-VMwareLM-ServiceVM-I17-H11-0-0-scsi-flat.vmdk 0 15G ./euclid-vcr/25/000025-import-0-0-scsi-flat.vmdk 0 15G ./euclid-vcr/26/000026-import-0-0-scsi-flat.vmdk 5.7G 20G ./euclid-vcr/27/000027-import-0-0-scsi-flat.vmdk 2.4G 15G ./euclid-vcr/28/000028-import-0-0-scsi-flat.vmdk 2.4G 15G ./euclid-vcr/29/000029-import-0-0-scsi-flat.vmdk 360M 15G ./euclid-vcr/30/000030-import-0-0-scsi-flat.vmdk 8.2G 20G ./euclid-vcr/31/000031-import-0-0-scsi-flat.vmdk 360M 15G ./euclid-vcr/32/000032-import-0-0-scsi-flat.vmdk 5.3G 12G ./euclid-vcr/33/000033-import-0-0-scsi-flat.vmdk 5.8G 20G ./euclid-vcr/34/000034-import-0-0-scsi-flat.vmdk 8.4G 40G ./euclid-vcr/35/000035-import-0-0-scsi-flat.vmdk 4.8G 12G ./euclid-vcr/36/000036-import-0-0-scsi-flat.vmdk 1.0G 1.0G ./euclid-vcr/36/000036-import-0-1-scsi-flat.vmdk 2.5G 25G ./euclid-vcr/37/000037-import-0-0-scsi-flat.vmdk 2.5G 25G ./euclid-vcr/38/000038-import-0-0-scsi-flat.vmdk 5.5G 12G ./euclid-vcr/39/000039-import-0-0-scsi-flat.vmdk 1.0G 1.0G ./euclid-vcr/39/000039-import-0-1-scsi-flat.vmdk 222M 10G ./euclid-vcr/40/000040-import-0-0-scsi-flat.vmdk 0 10G ./euclid-vcr/41/000041-import-0-0-scsi-flat.vmdk 13G 40G ./euclid-vcr/42/000042-import-0-0-scsi-flat.vmdk 261M 15G ./euclid-vcr/43/000043-import-0-0-scsi-flat.vmdk 7.1G 12G ./euclid-vcr/44/000044-import-0-0-scsi-flat.vmdk 9.5G 40G ./euclid-vcr/45/000045-import-0-0-scsi-flat.vmdk 7.1G 12G ./euclid-vcr/46/000046-import-0-0-scsi-flat.vmdk 2.9G 8.0G ./euclid-vcr/47/000047-import-1-0-scsi-flat.vmdk 245M 1.0G ./euclid-vcr/47/000047-import-0-0-scsi-flat.vmdk 245M 1.0G ./euclid-vcr/47/000047-import-0-1-scsi-flat.vmdk 2.6G 8.0G ./euclid-vcr/47/000047-import-1-1-scsi-flat.vmdk 3.3G 8.0G ./euclid-vcr/48/000048-import-1-0-scsi-flat.vmdk 245M 1.0G ./euclid-vcr/48/000048-import-0-0-scsi-flat.vmdk 245M 1.0G ./euclid-vcr/48/000048-import-0-1-scsi-flat.vmdk 2.9G 8.0G ./euclid-vcr/48/000048-import-1-1-scsi-flat.vmdk 6.9G 20G ./euclid-vcr/49/000049-import-0-0-scsi-flat.vmdk 3.8G 15G ./euclid-vcr/50/000050-import-0-0-scsi-flat.vmdk 2.7G 15G ./euclid-vcr/51/000051-import-0-0-scsi-flat.vmdk 2.7G 15G ./euclid-vcr/52/000052-import-0-0-scsi-flat.vmdk 834M 15G ./euclid-vcr/53/000053-import-0-0-scsi-flat.vmdk 8.3G 20G ./euclid-vcr/54/000054-import-0-0-scsi-flat.vmdk 4.8G 12G ./euclid-vcr/55/000055-import-0-0-scsi-flat.vmdk 1.0G 1.0G ./euclid-vcr/55/000055-import-0-1-scsi-flat.vmdk 2.4G 15G ./euclid-vcr/56/000056-import-0-0-scsi-flat.vmdk 2.7G 25G ./euclid-vcr/57/000057-import-0-0-scsi-flat.vmdk 360M 15G ./euclid-vcr/58/000058-import-0-0-scsi-flat.vmdk 11G 12G ./euclid-vcr/59/000059-import-0-0-scsi-flat.vmdk 5.5G 12G ./euclid-vcr/60/000060-import-0-0-scsi-flat.vmdk 1.0G 1.0G ./euclid-vcr/60/000060-import-0-1-scsi-flat.vmdk 261M 15G ./euclid-vcr/61/000061-import-0-0-scsi-flat.vmdk 5.8G 20G ./euclid-vcr/62/000062-import-0-0-scsi-flat.vmdk 8.5G 40G ./euclid-vcr/63/000063-import-0-0-scsi-flat.vmdk 7.1G 12G ./euclid-vcr/64/000064-import-0-0-scsi-flat.vmdk 3.0G 8.0G ./euclid-vcr/65/000065-import-1-0-scsi-flat.vmdk 245M 1.0G ./euclid-vcr/65/000065-import-0-0-scsi-flat.vmdk 245M 1.0G ./euclid-vcr/65/000065-import-0-1-scsi-flat.vmdk 2.7G 8.0G ./euclid-vcr/65/000065-import-1-1-scsi-flat.vmdk 223M 10G ./euclid-vcr/66/000066-import-0-0-scsi-flat.vmdk 1.0M 10G ./euclid-vcr/67/000067-import-0-0-scsi-flat.vmdk 3.8G 15G ./euclid-vcr/68/000068-import-0-0-scsi-flat.vmdk 245M 1.0G ./euclid-vcr/69/000069-import-0-1-scsi-flat.vmdk 245M 1.0G ./euclid-vcr/69/000069-import-0-0-scsi-flat.vmdk 1004M 8.0G ./euclid-vcr/69/000069-import-1-0-scsi-flat.vmdk 379M 15G ./euclid-vcr/544/000544-import-0-0-scsi-flat.vmdk 4.8G 12G ./euclid-vcr/545/000545-import-0-0-scsi-flat.vmdk 1.0G 1.0G ./euclid-vcr/545/000545-import-0-1-scsi-flat.vmdk 18G 40G ./euclid-vcr/546/000546-import-0-0-scsi-flat.vmdk 18G 40G ./euclid-vcr/553/000553-import-0-0-scsi-flat.vmdk 3.5G 20G ./euclid-vcr/554/000554-import-0-0-scsi-flat.vmdk 14G 40G ./euclid-vcr/555/000555-import-0-0-scsi-flat.vmdk 0 23G ./euclid-vcr/555/000555-import-0-1-scsi-flat.vmdk 14G 40G ./euclid-vcr/839/000839-import-0-0-scsi-flat.vmdk 0 23G ./euclid-vcr/839/000839-import-0-1-scsi-flat.vmdk 4.5G 20G ./euclid-vcr/838/000838-import-0-0-scsi-flat.vmdk 18G 40G ./euclid-vcr/837/000837-import-0-0-scsi-flat.vmdk 18G 40G ./euclid-vcr/836/000836-import-0-0-scsi-flat.vmdk 4.8G 12G ./euclid-vcr/835/000835-import-0-0-scsi-flat.vmdk 1.0G 1.0G ./euclid-vcr/835/000835-import-0-1-scsi-flat.vmdk 3.4G 15G ./euclid-vcr/834/000834-import-0-0-scsi-flat.vmdk 0 15G ./euclid-vcr/3246/003246-ESXi_41-2-0-scsi-flat.vmdk 834M 15G ./euclid-vcr/3246/003246-import-0-0-scsi-flat.vmdk 0 10G ./euclid-vcr/3246/003246-ESXi_41-1-0-scsi-flat.vmdk 40G 40G ./Celerra/Celerra-flat.vmdk Thin: 354G total Thick: 1.3T total
In this LabManager environment we moved from being over 95% utilization to being way under 50%. Specifically, all the datastores together were using 3700 GB and after the conversion we were using 1183 GB, that is a reduction of 68% of expensive shared SCSI storage utilization. Now, if you use this and also make savings, please consider donating (see right panel of http://vmutils.blogspot.com).
If you have some vDisks converted and others not, it may look like the following
# computeThin vCR_Production_CX3/ Thin Thick File 2.7G 10G vCR_Production_CX3/lm-temp/8017/008017-ESX4-flat.vmdk 295M 2.0G vCR_Production_CX3/lm-temp/8016/008016-ESX3i-flat.vmdk 2.1G 10G vCR_Production_CX3/lm-temp/8015/008015-ESX3-flat.vmdk 11G 12G vCR_Production_CX3/lm-temp/8014/008014-import-0-0-scsi-flat.vmdk 2.4G 15G vCR_Production_CX3/lm-temp/8013/008013-import-0-0-scsi-flat.vmdk 6.7G 40G vCR_Production_CX3/lm-temp/8012/008012-import-0-0-scsi-flat.vmdk [...] 20G 20G vCR_Production_CX3/lm-temp/8853/008853-import-0-1-scsi-flat.vmdk 12G 12G vCR_Production_CX3/lm-temp/11396/011396-import-0-0-scsi-flat.vmdk 15G 15G vCR_Production_CX3/lm-temp/11397/011397-import-0-0-scsi-flat.vmdk 40G 40G vCR_Production_CX3/lm-temp/11398/011398-import-0-0-scsi-flat.vmdk 40G 40G vCR_Production_CX3/lm-temp/11399/011399-import-0-0-scsi-flat.vmdk 15G 15G vCR_Production_CX3/lm-temp/11400/011400-import-0-0-scsi-flat.vmdk 40G 40G vCR_Production_CX3/lm-temp/11403/011403-import-0-0-scsi-flat.vmdk 40G 40G vCR_Production_CX3/lm-temp/11404/011404-import-0-0-scsi-flat.vmdk [...] Thin: 1.5T total Thick: 1.7T total
As explained above, this script calculates the savings ONCE you have converted the disks (note the difference between both sections), not before.
8. Recommendations
8.1. Running in parallel
If you want to convert the disks on many datastores at the same time, run ONE script per datastore. There is no point on stressing the datastore with multiple threads. Even if the array is very capable, you don't want 2 scripts trying to convert the same files to leave a mess behind.
If you think that I/O can be a bottleneck you can also run each one of those per-datastore-script from a different ESX.
Running in parallel is handy as you have to confirm every conversion and you don't want to spend the whole weekend looking at a putty.
8.2. Avoid fragmentation
If your datastores are really full (>95%), my recommendation is to create free space as soon as possible. That is, convert a disk and clean up, convert another disk and clean up,... Until you have some room on the datastore. How much is 'some room'? It depends of the size of the datastore and its files. I don't pretend to give an explanation about the theory of disk fragmentation, but I'd say until you are around 90%.
Then you can convert disks in groups, releasing more and more space after every group.
8.3. Verify before cleaning up
That is always my recommendation for the people who start using the script.
- Convert
- Verify, that is, power on your VM/LabManager configuration
- Clean up
The clean up can be done with the VM running, and in fact it is recommended, as if the VM is running, its vDisks cannot be deleted.
9. What happens if ... ?
9.1. What happens if I run makeThin against a VM that is running?
9.1.1. VM with snapshots
As explained above, if the VM has snapshots, the BaseDisk will not be locked (that is used for many backup products to read the vDisk while the VM is writing on the snapshot) and therefore you can rename the vDisk and then clone it back to the original name (this is what makeThin does).
This won't break the VM, but the VM will continue reading from the thick vDisk while it stays running and if you commit the snapshots they will commit to the thick vDisk. Once it has been powered off and turned back on it will read from the thin vDisk.
By all means you should avoid this to happen, as you may end up deleting the vDisk that has more up to date information.
Example below (VM has 3 vDisks, thick, thin and zeroed).
## Check locks on files
# file *vmdk
rg-test-empty-000001-delta.vmdk: VMware3 disk image 8389120 bytes
rg-test-empty-000001.vmdk: ASCII English text
rg-test-empty-000002-delta.vmdk: writable, regular file, no read permission
rg-test-empty-000002.vmdk: ASCII English text
rg-test-empty_1-000001-delta.vmdk: VMware3 disk image 13632000 bytes
rg-test-empty_1-000001.vmdk: ASCII English text
rg-test-empty_1-000002-delta.vmdk: writable, regular file, no read permission
rg-test-empty_1-000002.vmdk: ASCII English text
rg-test-empty_1-flat.vmdk: data
rg-test-empty_1.vmdk: ASCII English text
rg-test-empty_2-000001-delta.vmdk: VMware3 disk image 3146240 bytes
rg-test-empty_2-000001.vmdk: ASCII English text
rg-test-empty_2-000002-delta.vmdk: writable, regular file, no read permission
rg-test-empty_2-000002.vmdk: ASCII English text
rg-test-empty_2-flat.vmdk: data
rg-test-empty_2.vmdk: ASCII English text
rg-test-empty-flat.vmdk: data
rg-test-empty.vmdk: ASCII English text
## Only one disk is thin.
# grep -i thin rg-test-empty.vmdk rg-test-empty_[12].vmdk
rg-test-empty_2.vmdk:ddb.thinProvisioned = "1"
# findAndMakeDiskThin .
-----------------------------------------------------------------------------
Working with ./rg-test-empty.vmdk. Maximum space needed: 80M
Filesystem Size Used Avail Use% Mounted on
/vmfs/volumes/4c80e2b4-24c5bb66-9171-001a648f3ccc/rg-test-empty
270G 227G 42G 84% /vmfs/volumes/datastore1 (2)
Verify the file is not in use
./rg-test-empty-flat.vmdk: data
-- Convert to thin? (y/n): y
Will run: vmkfstools -E "./rg-test-empty.vmdk" "./SAFETMPrg-test-empty.vmdk"
Will run: vmkfstools -i "./SAFETMPrg-test-empty.vmdk" -d thin "./rg-test-empty.vmdk"
Destination disk format: VMFS thin-provisioned
Cloning disk './SAFETMPrg-test-empty.vmdk'...
Clone: 100% done.
Visual verification
-rw------- 1 root root 83886080 Jun 2 21:33 ./rg-test-empty-flat.vmdk
-rw------- 1 root root 518 Jun 2 21:33 ./rg-test-empty.vmdk
-rw------- 1 root root 83886080 Jun 2 21:21 ./SAFETMPrg-test-empty-flat.vmdk
-rw------- 1 root root 476 Jun 2 21:33 ./SAFETMPrg-test-empty.vmdk
-----------------------------------------------------------------------------
Working with ./rg-test-empty_1.vmdk. Maximum space needed: 90M
Filesystem Size Used Avail Use% Mounted on
/vmfs/volumes/4c80e2b4-24c5bb66-9171-001a648f3ccc/rg-test-empty
270G 227G 42G 84% /vmfs/volumes/datastore1 (2)
Verify the file is not in use
./rg-test-empty_1-flat.vmdk: data
-- Convert to thin? (y/n): y
Will run: vmkfstools -E "./rg-test-empty_1.vmdk" "./SAFETMPrg-test-empty_1.vmdk"
Will run: vmkfstools -i "./SAFETMPrg-test-empty_1.vmdk" -d thin "./rg-test-empty_1.vmdk"
Destination disk format: VMFS thin-provisioned
Cloning disk './SAFETMPrg-test-empty_1.vmdk'...
Clone: 100% done.
Visual verification
-rw------- 1 root root 94371840 Jun 2 21:34 ./rg-test-empty_1-flat.vmdk
-rw------- 1 root root 520 Jun 2 21:34 ./rg-test-empty_1.vmdk
-rw------- 1 root root 94371840 Jun 2 21:22 ./SAFETMPrg-test-empty_1-flat.vmdk
-rw------- 1 root root 478 Jun 2 21:34 ./SAFETMPrg-test-empty_1.vmdk
-----------------------------------------------------------------------------
./rg-test-empty_2.vmdk is thinProvisioned, skipping.
# computeThin .
Thin Thick File
80M 80M ./SAFETMPrg-test-empty-flat.vmdk
90M 90M ./SAFETMPrg-test-empty_1-flat.vmdk
0 70M ./rg-test-empty_2-flat.vmdk
0 80M ./rg-test-empty-flat.vmdk
0 90M ./rg-test-empty_1-flat.vmdk
Thin: 170M total
Thick: 410M total
## Two vDisks get converted, but the VM stays running, and reading now from the SAFETMP ones.
9.1.2. VM without snapshots
In this case, the BaseDisk will be locked, which won't allow the script to do the clone back, but it will allow it to rename the vDisk. The VM won't notice that until it uses that file name for any operation, like when you take snapshots, Vmotion, SVmotion, ...
This case is worse than the case before, because you have no clone of the disk and because any operation that makes the VM realize that the BaseDisk has changed name will bring it down.
## Check locks
# file *vmdk
rg-test-empty2_1-flat.vmdk: writable, regular file, no read permission
rg-test-empty2_1.vmdk: ASCII English text
rg-test-empty2_2-flat.vmdk: writable, regular file, no read permission
rg-test-empty2_2.vmdk: ASCII English text
rg-test-empty2-flat.vmdk: writable, regular file, no read permission
rg-test-empty2.vmdk: ASCII English text
## Only one is thin
# grep -i thin *vmdk
grep: rg-test-empty2_1-flat.vmdk: Device or resource busy
grep: rg-test-empty2_2-flat.vmdk: Device or resource busy
rg-test-empty2_2.vmdk:ddb.thinProvisioned = "1"
grep: rg-test-empty2-flat.vmdk: Device or resource busy
# findAndMakeDiskThin .
-----------------------------------------------------------------------------
Working with ./rg-test-empty2.vmdk. Maximum space needed: 80M
Filesystem Size Used Avail Use% Mounted on
/vmfs/volumes/4c80e2b4-24c5bb66-9171-001a648f3ccc/rg-test-empty2
270G 228G 41G 84% /vmfs/volumes/datastore1 (2)
Verify the file is not in use
./rg-test-empty2-flat.vmdk: writable, regular file, no read permission
-- Convert to thin? (y/n): y
Will run: vmkfstools -E "./rg-test-empty2.vmdk" "./SAFETMPrg-test-empty2.vmdk"
Will run: vmkfstools -i "./SAFETMPrg-test-empty2.vmdk" -d thin "./rg-test-empty2.vmdk"
Destination disk format: VMFS thin-provisioned
Failed to open './SAFETMPrg-test-empty2.vmdk': Failed to lock the file (16392).
Visual verification
ls: ./rg-test-empty2.vmdk: No such file or directory
ls: ./rg-test-empty2-flat.vmdk: No such file or directory
-rw------- 1 root root 83886080 Jun 2 21:22 ./SAFETMPrg-test-empty2-flat.vmdk
-rw------- 1 root root 500 Jun 2 21:39 ./SAFETMPrg-test-empty2.vmdk
-----------------------------------------------------------------------------
The visual verification is not positive. The file got renamed but couldn't do the clone as it is locked. The VM stays running using SAFETMPrg-test-empty2.vmdk
Working with ./rg-test-empty2_1.vmdk. Maximum space needed: 90M
Filesystem Size Used Avail Use% Mounted on
/vmfs/volumes/4c80e2b4-24c5bb66-9171-001a648f3ccc/rg-test-empty2
270G 228G 41G 84% /vmfs/volumes/datastore1 (2)
Verify the file is not in use
./rg-test-empty2_1-flat.vmdk: writable, regular file, no read permission
-- Convert to thin? (y/n): y
Will run: vmkfstools -E "./rg-test-empty2_1.vmdk" "./SAFETMPrg-test-empty2_1.vmdk"
Will run: vmkfstools -i "./SAFETMPrg-test-empty2_1.vmdk" -d thin "./rg-test-empty2_1.vmdk"
Destination disk format: VMFS thin-provisioned
Failed to open './SAFETMPrg-test-empty2_1.vmdk': Failed to lock the file (16392).
Visual verification
ls: ./rg-test-empty2_1.vmdk: No such file or directory
ls: ./rg-test-empty2_1-flat.vmdk: No such file or directory
-rw------- 1 root root 94371840 Jun 2 21:22 ./SAFETMPrg-test-empty2_1-flat.vmdk
-rw------- 1 root root 502 Jun 2 21:39 ./SAFETMPrg-test-empty2_1.vmdk
-----------------------------------------------------------------------------
./rg-test-empty2_2.vmdk is thinProvisioned, skipping.
# computeThin .
Thin Thick File
80M 80M ./SAFETMPrg-test-empty2-flat.vmdk
90M 90M ./SAFETMPrg-test-empty2_1-flat.vmdk
0 70M ./rg-test-empty2_2-flat.vmdk
Thin: 170M total
Thick: 240M total
## Two files are still thick, the other was thin initially, but we are missing 2 clones.
# ll *vmdk
-rw------- 1 root root 73400320 Jun 2 21:22 rg-test-empty2_2-flat.vmdk
-rw------- 1 root root 521 Jun 2 21:23 rg-test-empty2_2.vmdk
-rw------- 1 root root 94371840 Jun 2 21:22 SAFETMPrg-test-empty2_1-flat.vmdk
-rw------- 1 root root 502 Jun 2 21:39 SAFETMPrg-test-empty2_1.vmdk
-rw------- 1 root root 83886080 Jun 2 21:22 SAFETMPrg-test-empty2-flat.vmdk
-rw------- 1 root root 500 Jun 2 21:39 SAFETMPrg-test-empty2.vmdk
#
9.2. What happens if I power on my VM while makeThin is converting its vDisk?
If makeThin is still clonning the vDisk, the VM won't power on as it cannot open and lock the file because the cloning operation locks it.
See here the error message you would get.
 |
If the VM has snapshots, the answer gets more complicated as there are many possible situations.
9.3. What happens if I close putty/SSH session while the script is running?
The same as when you press Ctrl+C.
Now, it depends where on the code the interruption came.
The part of renaming the disks is very fast, so it gets done or not.
The part of cloning the vDisk is what takes most of the process. After doing some research, the conclusion is that the destination file, even though it is created and has the same size (that is the first step of the clonning process), it won't contain the same information as the source.
You can use the commands cmp and hexdump of Linux to verify it. I had a good time researching this.
# vmkfstools -i ../VM_Cisco_Tools/VM_Cisco_Tools.vmdk stop60percent.vmdk Destination disk format: VMFS zeroedthick Cloning disk '../VM_Cisco_Tools/VM_Cisco_Tools.vmdk'... Clone: 57% done. ## I pressed Ctrl+C as close to 60% as I could. # cmp ../VM_Cisco_Tools/VM_Cisco_Tools-flat.vmdk stop60percent-flat.vmdk ../VM_Cisco_Tools/VM_Cisco_Tools-flat.vmdk stop60percent-flat.vmdk differ: byte 3120562177, line 969995 # python >>> 3120562177.0 / (1024*1024*1024) 2.9062500009313226 ## 2.9 GB is more or less 60% of the 5GB size of the original file. ## Note that if the source file is fairly empty, the clonning would go very fast and this number can be misleading. ## You can also open both files on 2 consoles with "hexdump -C -s XXXX file-flat.vmdk | less" where XXXX is some bytes before the point where they "differ: byte 3120562177" (reduce the 4th digit from right). Once you reach the differ point you would not be able to continue looking to the content, while on the original file you can continue without problem.
9.4. What happens if my VMs have RDMs?
The script won't touch the RDMs or its vmdks.
10. FAQ
10.1. How do I know the location of the VMs in my LabManager configuration?
In LabManager, open the configuration, and go to properties on each VM, you will get a number ID.
 |
Then search for that number on the ESX and you will know its location.
Example:
# find /vmfs/volumes/ -name 9535 /vmfs/volumes/4ca0591a-e574cc1e-bfd8-0024817d6e58/vtrain/9535
10.2. My VM has many snapshots, how do I know where the BaseDisks are?
makeThin does look for vDisks to convert recursively in the locations indicated. It does not follow chains of snapshots looking for the BaseDisk (I did SnapVMX long time ago for that), as doing so could take the script out of the locations indicated by the user (even to a different datastore).
See below a case where we find the BaseDisk easily.
First we access the datastore where we have previously copied the code of SnapVMX
# cd /vmfs/volumes/vCR_Production_CX4/
And load the functions of SnapVMX (same procedure that for makeThin)
# . SnapVMX.source.code.txt
Then we go to the location of the LabManager VM as indicated in LabManager and run SnapVMX against the vmx file. If you want to run it agains a vmdk use SnapTree.
# cd lm-temp/8885/ # SnapVMX 008885-ESX3.vmx Base Disk: 008885-import-0-0-scsi.vmdk Size: 30G ----------------
This first VM had no snapshots, so lets try another one ...
# cd ../8907/
# SnapVMX 008907-Dom.Controller.vmx
008907-import-0-0-scsi.vmdk Size: 65M
/vmfs/volumes/4c3c2a43-f68871d3-fbc8-002564fb7c46/lm-temp/7588/007588-import-0-0-scsi.vmdk Size: 177M
/vmfs/volumes/4c3c2a43-f68871d3-fbc8-002564fb7c46/lm-temp/7351/007351-import-0-0-scsi.vmdk Size: 65M
/vmfs/volumes/4c3c2a43-f68871d3-fbc8-002564fb7c46/lm-temp/7336/007336-import-0-0-scsi.vmdk Size: 49M
Base Disk: /vmfs/volumes/4c3c2a43-f68871d3-fbc8-002564fb7c46/lm-temp/7289/007289-import-0-0-scsi.vmdk Size: 12G
Space needed on this Datastore to delete all the snapshots of this disk is: 608.15 Megabytes = 0.59 Gigabytes
----------------
008907-import-0-1-scsi.vmdk Size: 4.0K
/vmfs/volumes/4c3c2a43-f68871d3-fbc8-002564fb7c46/lm-temp/7588/007588-import-0-1-scsi.vmdk Size: 4.0K
/vmfs/volumes/4c3c2a43-f68871d3-fbc8-002564fb7c46/lm-temp/7351/007351-import-0-1-scsi.vmdk Size: 4.0K
/vmfs/volumes/4c3c2a43-f68871d3-fbc8-002564fb7c46/lm-temp/7336/007336-import-0-1-scsi.vmdk Size: 4.0K
Base Disk: /vmfs/volumes/4c3c2a43-f68871d3-fbc8-002564fb7c46/lm-temp/7289/007289-import-0-1-scsi.vmdk Size: 1.0G
Space needed on this Datastore to delete all the snapshots of this disk is: 0.02 Megabytes = 0.00 Gigabytes
----------------
The line with Base Disk indicates where it is. Now you can go to that location and run makeThin. You could also go to the upper folder and run "# makeThin .", it would find it anyway. The difference is that it may find many other vDisks and it will ask you for each one.
10.3. I run the script but nothing happens, why?
Example:
# findAndMakeDiskThin /vmfs/volumes/SRT-Delivery-Production-1/lmburtrn/6270
User says: I expected it to work, and verified the VM id in Lab Manager. Nothing happens just a prompt. I expect it to work normally like the case below
# findAndMakeDiskThin /vmfs/volumes/SRT-Delivery-Production-1/lmburtrn/5323 ----------------------------------------------------------------------------- /vmfs/volumes/SRT-Delivery-Production-1/lmburtrn/5323/005323-import-0-0-scsi.vmdk is thinProvisioned, skipping.
Explanation:
It does nothing because there is nothing to do. The disk on that folder was a snapshot.
The disk didn't pass the 1st row of checks, that's why you didn't receive any feedback.
Imagine you run the script on the parent folder (in a LabManager environment), you would get a line for every single snapshot-disk that we will not even look at, too much noise.
Imagine a VM with 15 snapshots, and 4 vDisks , you would get 60 lines saying "this is a snapshot", too much noise.
11. Acknowledgement
The author would like to thank Antonio Allegue Leira for his revision of this document.
The author would also like to thank his team members for their suggestions and questions that resulted in a better program.
12. About the author
This program and the documentation was developed by Ruben Miguelez Garcia (also known as Ruben Garcia) on 2010/2011. Ruben enjoys writing tiny programs and/or doing modification to systems, making computers/systems work for humans, saving those humans lots of time and saving companies lots of money. He did that on every company he has been, even while he was at university.
You can find Ruben on Mr.Ruben.Garcia@gmail.com
Challenging questions/projects and suggestions for improvement of the documentation/program are welcome.
Donations and gifts accepted :o)
13. References
- SnapVMX http://vmutils.blogspot.com
- Thin-Provisioning http://www.VMware.com/files/pdf/VMware-vStorage-Thin-Provisioning-DS-EN.pdf
14. Source code
#-------------------------------------------------------------------------
# Copyright (C) 2010, 2011 Ruben Miguelez Garcia
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, version 3.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
#-------------------------------------------------------------------------
#
# The documentation of this program can be found on http://vmutils.blogspot.com/
#
# To start using this script, place this file on an ESX and run
# . /path/<name_of_this_file>
#
#
# v1 All basic functionality working perfectly
# v2 Improved code to extract the vmdks from the vmx instead of just try all vmdks
# v3 Improved code of cleanup to not ask when there is nothing to delete
# v4 Added functionality to pass as a parameter datastores to ignore during the checks (if desired)
# v5 Reduced complexity of entire program, as clonning never touches the CID and therefore there is no need to look for other vmdks pointing to the one we will convert.
# v5 Added functions to convert disks alone, find them and do both (find+convert)
# v5 Added checks agains RDMs
# v6 Cleaned up unnecessary functions and added computeThin
#
# Every time a pipe is used the second process may be executed in a subshell. If so, the keyboard redirection does not work
# For that reason some of these functions can not be used in other scripts.
#
#==== Auxiliary Functions ====================================================
SizeH () { ls -lh "$1" | awk '{ print $5 }'; }
#------------------
isThin () { result=`grep -i thinProvisioned "$1"` ; if [ "$result" = 'ddb.thinProvisioned = "1"' ] ; then echo yes; else echo no; fi; }
#------------------
isBaseDisk () {
# Ensure that the disk is both, base disk (no snapshot) and a virtual disk (not an RDM)
result=`grep -i parentCID "$1"`;
result2=`grep -i createType "$1"`;
if [ "$result" = "parentCID=ffffffff" -a "$result2" = 'createType="vmfs"' ] ; then echo "yes"; else echo "no"; fi; }
#============= cleanupSAFETMP =================================================
cleanupSAFETMP () {
# Usage: cleanupSAFETMP /path/
# Find SAFETMP*vmdk files recursively under the path specified and ask you for confirmation before deleting them.
# If no argument provided, then look where we are
if [ "$1" != "" ]; then LOCATION="$*"; else LOCATION="."; fi;
# Check if there is something to clean up.
RESULTLOOKINGTARGETS=`find $LOCATION -name "SAFETMP*vmdk"`;
# Decide
if [ "$RESULTLOOKINGTARGETS" = "" ] ; then
echo -e "\nNo SAFETMP*vmdk files found under the specified location (default is \".\"). Nothing to clean up. Exiting.\n";
else
echo -e "\n\tFiles found:";
echo "$RESULTLOOKINGTARGETS";
echo -e "\nDo you want to delete these files?\n";
ANSWER="0"; read -p "Answer (y/n): " -n 1 ANSWER;
if [ "$ANSWER" = "Y" -o "$ANSWER" = "y" ] ; then
find $LOCATION -name "SAFETMP*vmdk" ! -name "*-flat.vmdk" ! -name "*-delta.vmdk" | while read TODELETE ; do
vmkfstools -U "$TODELETE" ; done;
echo -e "\nDone.\n";
else
echo -e "\nNothing deleted.\n";
fi;
fi;
}
#============ findBaseDisks ======================================================
findBaseDisks () {
# Easy query to find vmdks that have a -flat.vmdk and are supposed to be BaseDisks.
# Snapshots have a -delta.vmdk supporting the data instead.
# This only finds Base Disks that have BOTH files, file.vmdk and its file-flat.vmdk
# Get original location
ORIGINALLOCATION=`pwd`;
# If no argument provided, then look where we are
if [ "$1" != "" ]; then
LOCATION="$*";
else
LOCATION=".";
fi;
# Find the -flat.vmdk files and from them the .vmdk and display if it exists.
find $LOCATION -iname "*-flat.vmdk" | while read DISKPATH; do
DPATH=`dirname "$DISKPATH"`;
DNAME=`basename "$DISKPATH" "-flat.vmdk"`;
if [ -e "$DPATH/$DNAME.vmdk" ]; then echo "$DPATH/$DNAME.vmdk"; fi;
done; }
#============= findAndMakeDiskThin = makeThin ======================================
alias makeThin=findAndMakeDiskThin ;
findAndMakeDiskThin () {
# Usage: findAndMakeDiskThin /path/
# It will find Base Disks recursively under the path specified and one by one, convert them to thin provision ONLY if required and ONLY if you confirm the operation.
# To say YES you just need to press y or Y, without Enter.
# Any other key, including Enter will mean NO.
{ # Another pair of keys to allow keyboard redirection
# Get original location
ORIGINALLOCATION=`pwd`;
# If no argument provided, then look where we are
if [ "$1" != "" ]; then
LOCATION="$*";
else
LOCATION=".";
fi;
findBaseDisks "$LOCATION" | while read VDISK ; do
# Check it exists
if [ ! -e "$VDISK" ]; then echo "$VDISK file not found."; else
echo -e "\n-----------------------------------------------------------------------------\n"
# Check that the disk is thin before continuing
if [ "`isThin \"$VDISK\"`" = "yes" ]; then echo "$VDISK is thinProvisioned, skipping." ; else
# Check that the disk is a Base Disk (not snapshot/RDM) before continuing
if [ "`isBaseDisk \"$VDISK\"`" = "no" ]; then echo "$VDISK is a snapshot and/or an RDM, skipping." ; else
# Display its size and free space on Datastore
DPATH=`dirname "$VDISK"`;
DNAME=`basename "$VDISK" ".vmdk"`;
DNAMEFLAT="$DNAME-flat.vmdk";
echo -en "\nWorking with $VDISK. Maximum space needed: "; SizeH "$DPATH/$DNAMEFLAT"; echo "";
vdf -h "$DPATH";
# Verification of locks on file
echo -e "Verify the file is not in use"
file "$DPATH/$DNAMEFLAT"
# Ask for confirmation
echo -en "\n-- Convert to thin? (y/n): "; INPUT="0"; read -n 1 INPUT <&6; if [ "$INPUT" = "Y" -o "$INPUT" = "y" ] ; then
echo ""; #echo -e "yes do it\n" ;
# Rename source
echo "Will run: vmkfstools -E \"$VDISK\" \"$DPATH/SAFETMP$DNAME.vmdk\""
vmkfstools -E "$VDISK" "$DPATH/SAFETMP$DNAME.vmdk"
# Clone to thin
echo "Will run: vmkfstools -i \"$DPATH/SAFETMP$DNAME.vmdk\" -d thin \"$VDISK\""
vmkfstools -i "$DPATH/SAFETMP$DNAME.vmdk" -d thin "$VDISK"
# Verify
echo -e "\n Visual verification"
ls -l "$DPATH/SAFETMP$DNAME.vmdk" "$DPATH/SAFETMP$DNAME-flat.vmdk" "$DPATH/$DNAME.vmdk" "$DPATH/$DNAME-flat.vmdk"
else
#echo "DON'T do it" ;
echo ""
fi; # Confirmation to convert
fi; # is BaseDisk
fi; # isThin
fi; # file exist
done;
} 6>&0 # for redirection of keyboard
echo "";
};
#============ computeThin =================================================
computeThin () {
# Usage: computeThin /path/
# Using the linux command 'du' it will calculate the difference between real size and apparent size in terms of blocks used.
# The comparison is valid once the file is thin provision, otherwise both results are the same.
LOCATION=$1;
(echo "Thin Thick File"; find "$LOCATION" -name "*-flat.vmdk" | while read DISK; do paste <(du -h "$DISK" | awk '{print $1}' ) <(du -h --apparent-size "$DISK" ) ; done ) | column -t;
echo "";
(echo -n "Thin: " ; find "$LOCATION" -name "*-flat.vmdk" -print0 | xargs -0 du -h -c | tail -1 ; echo -n "Thick: " ; find "$LOCATION" -name "*-flat.vmdk" -print0 | xargs -0 du -h --apparent-size -c | tail -1 ) | column -t;
}
#=============================================================================