Skip to main content

Preparing Linux for optimal thin provisioning and deduplication

VMware and disk-array based thin provisioning can help you economize on disk space, just as disk-array based deduplication can. But how to take optimal advantage of those techniques ? Executive summary:
dd if=/dev/zero of=/tmp/tempfile.zeroes ; rm -f /tmp/tempfile.zeroes

First of all, both thin provisioning and deduplication work transparently. But you can make them work better with a little bit of work. There's three different types of blocks on your (virtual) disk that we need to distinguish: data blocks, old data blocks, and empty blocks.
Data blocks and empty blocks are what they are, and can't be influenced. Data blocks contain data for files that are on your filesystem. Empty blocks are empty, have never been written to with any real data, so their blank, contain only zeroes.
It's the "old data" blocks that we can improve: they contain data that used to be part of a file. The file got deleted, but the contents are still there.
Overwriting those blocks with zeroes will help a (re)conversion to thin provisioning later, and for dedup they are now empty blocks again, so perfect sharing possibilities.
Overwriting just the old data blocks is hard, but you can probably live with overwriting both empty and old data blocks. This will allocate all blocks (byebye thin provisioning, for now), but the contents will be all-zeroes: perfect thin re-thin-provisioning later, and perfect for dedup.
for each local filesystem mounted on your system, you can execute
dd if=/dev/zero of=/$mountpoint/tempfile.zeroes ; rm -f /$mountpoint/tempfile.zeroes

be careful, this will - very briefly - fill up your filesystem(s). If your application can't handle this, do it when that app isn't running. For those of you who use LVM, the unallocated PE's in every VG aren't cleared by this procedure, so for every Volume Group, find out name and available free space:
vgs -o vg_name,vg_free

then run (fill $vg_free and $vg_name with the data you just gathered)
lvcreate -n zerolv -L $vg_free $vg_name && dd if=/dev/zero of=/dev/$vgname/zerolv ; lvchange -a n /dev/$vgname/zerolv && lvremove /dev/$vgname/zerolv


localhost said…
This reminds me of a blog post I wrote a year ago: (in Dutch)
bert said…
Forgot to mention: for windows the tool of choice is sdelete.
Jonathan Barber said…
Many thanks for the post, very helpful to know that you have to use the same procedure for Linux as for Windows.

Just to let you know that you can create a LV that fills a VG with the "-l 100%FREE" argument to lvcreate. e.g:
lvcreate -l 100%FREE -n zerolv $vg_name

Popular posts from this blog

Volkswagen UHV bluetooth touch adapter & its problems

My Volkswagen car has the "universal cellphone preparation" UHV built-in. This is the main part of a car kit, but requires an additional adapter for connecting to a cellphone. At first, I was using an adapter for my good old Nokia 6310, even after I changed to the Nokia E71. Connecting was easy: pair the phone with the "VW UHV" bluetooth entity, and done. This has the phone connected to the car kit at all times, so even non-call-related functions use the car audio system (e.g. voice recognition).
But progress will have its way, no matter what happens. So in comes the "bluetooth touch adapter". Instead of a phone-specific adapter, this is a small touchscreen device that slots into the UHV dashboard mount. Connecting a phone is very different now:
the Bluetooth Touch Adapter connects to the "VW UHV" device via bluetooth
the phone connects to "Touch Adapter" device, also via bluetoothThe device doesn't allow step 2 if step 1 didn't s…

How to solve "user locked out due to failed logins" in vSphere vMA

In vSphere 6, if the vi-admin account get locked because of too many failed logins, and you don't have the root password of the appliance, you can reset the account(s) using these steps:

reboot the vMAfrom GRUB, "e"dit the entry"a"ppend init=/bin/bash"b"oot# pam_tally2 --user=vi-admin --reset# passwd vi-admin # Optional. Only if you want to change the password for vi-admin.# exitreset the vMAlog in with vi-admin These steps can be repeated for root or any other account that gets locked out.

If you do have root or vi-admin access, "sudo pam_tally2 --user=mylockeduser --reset" would do it, no reboot required.

Multiple VLANs on a Synology NAS

Synology, like other SOHO/SMB NAS vendors, touts VLAN functionality with their current DSM 4.1 software. However, the web interface just lets you specify one VLAN tag to use over each eth interface (or bond interface).
Manual approachIn the busybox environment that you can ssh into as root (after enabling ssh through the webinterface), there's all the tools you need to use multiple VLANs over one link (eth or bond), however:
First you insert the 802.1q module into the Linux kernel:
 /sbin/lsmod | /bin/grep -q 8021q || /sbin/insmod /lib/modules/8021q.koThen you add each VLAN you need to every interface (bond0 in this example)
 /sbin/vconfig add bond0 4And finally you can configure IP addresses on every interface.vlan combination (bond0.4 in this example)
 /sbin/ifconfig bond0.4 broadcast netmask same type of script would work on a QNAP NAS too, by the way. They offer 8021q.ko and vconfig in their commandline environment as well.
Packets from…