I’ve grown to love ZFS. But using on Virtual Machines did cause me to learn a new trick. Virtual Machines allow you lously allocate disk space which can be very helpful but managing this doesn’t always play nice with ZFS.
Virtual Machines and Thin Provisioning
Virtual Machine products usually allow a feature that called “Thin Provisioning” under the VMware family of products. In detail, when you provision a “disk drive” for your VM, you can just designate the size of the drive without actually claiming any space for it. So, you can thin provision a 120GB drive and only pull a few kilobytes from the filesystem for the hypervisor to manage things. This works because the drive will return a set of zeros when you read a sector that hasn’t been previously written. This is all built on a Unix feature called Sparse files. In Unix, a sparse file behaves exactly this way. If you read a block in the file that hasn’t ever been written to, you get zeros without the Unix having to do anything. When you write that sector, the block is actually saved to the disk and the next time you do a read, you’ll get what you wrote last time. In VMware this is handy because your VM’s disk drive only stores the stuff that you’ve written.
Day to day use
As you use this thinly provisioned drive you’ll write data and the hypervisor will store it and all work work great. What’s interesting is what happens when you delete a file. Deleting a file won’t zero the sectors that it use, and that wouldn’t matter any how because a sparse file in Unix differentiates between a block of zeros in an unwritten portion of the file and a block that you filled with zeros through a write operation. The second block of zeros actually takes up space. It’s important to note that as far as the hypervisor is concerned, a block of zeros in a thin provisioned drive never needs to take up space.
Reclaiming space
After a period of use, a normal filesystem will have a bunch of block that are written, and filled with contents that used to be a part of files that have since been deleted. If the disk drive that we’re talking about is a sparse file on a VMware server this file will take up more and more space up until you actually “fill up the drive”. A standard method for reclaiming this space on older filesystems was to overwrite the entire unallocated part of the drive with zeros and ask VMware to re-examine the virtual drive, punching holes where any large segment of zeros is found.
Where ZFS fits in
Older filesystems like UFS and EXT4 generally don’t support advanced operations like automatic filesystem object compression and deduplication. On these systems writing a block of zeros to a file will generally actually write a block of zeros to the disk drive. On ZFS, where compression and deduplication are available, writing a block of zeros need not write any data at all. In effect you can’t write a block of zeros to a file on a host using zfs.
Reclaiming space normally
To reclaim space from EXT4 or UFS the general procedure is simple. Within the running VM:
## Shut the system down to single user mode # for fs in <mounted-filesystems>; do > dd if=/dev/zero of="${fs}/.zerofill" bs=26214400 > rm "${fs}/.zerofill" > done ## Now all the unallocated space should have been zeroed out ## shutdown the VM
Now on the host run whatever tool you need to reclaim space from the VM. This usually involves having the hypervisor or a utility read the virtual disk block by block discarding and blocks that are completely filled with zeros. In VMware Fusion, this is the utility:
# vmware-vdiskmanager -k <virtual-disk-file>
It’s usually that simple. When the filesystem is ZFS, things are different. For one thing, several disk mounting points will exist as a part of a zfs pool. You only need to fill one “filesystem” on a pool to zero out all of the unallocated space. More importantly though with compression and dedup on, you won’t be able to write zeros to the virtual drive. Here’s the procedure on a machine with zfs:
## Again, shut the system down to single user mode. # for p in <affected-zfs-pools>; do > zfs set compression=off,dedup=off "${p}" > dd if=/dev/zero of=<pool-mountpoint>/.zerofill bs=26214400 > rm <pool-mountpoint/.zerofill > zfs set compression=on,dedup=on > done # shutdown -h now
At this point the machine you should be able to reclaim the zeroed space using the hypervisors tools as before.