the birdhouse in my soul

In the aftermath of the previous post, I ended up monitoring the progress of several huge pvmove and md RAID1 resync operations. Running "vmstat 1" however only shows pvmove block I/O activity, but not md RAID1 resyncing. Fortunately, "dstat -d" shows the reads and writes of both pvmove and md RAID1 resync. ("-d" is automatically included in "dstat -a" by the way.) N.B. This only affects block I/O measurements, you'll still see the impact on interrupts and CPU statistics in vmstat.

When the SATA standard was introduced in PCs, I'd read that the electrical connections of both the data and the power connector had been designed with hotplug in mind. But just as with many hotplug-able technologies, I never actually tried it (hotplug PCI, anyone ?). Until now, that is. Job at hand: replace the software-mirrored SATA drives in high-end Dell workstations with bigger ones. Without losing the data, of course. A perfect opportunity to test how Linux handles SATA hotplug ! Actually, adding the drive was a breeze, Linux automatically detects the drive, and I could sfdisk, and mdadm --add. Next step was hot-removing the old drives: mdadm --fail and mdadm --remove, then physically unplug the drives. I didn't expect it to be so easy, to be honest. Great technology !

The familiar packaging of a product gets replaced (or upgraded), the old packaging is no longer available, and the new packaging is priced almost double the original. Example of the week: 20cl cream at Carrefour. Old brick container, <40c. New brick container with "easy screw-off opening": 69c. The old packaging must have been easier to recycle and cheaper to make, right ?

I had used audacity before, to record a couple of old audio cassettes, cut them into tracks, and burn them on CD (and convert them to .ogg or .mp3). But that was years ago. Last week, I needed audacity again, and had to crawl out of two pitfalls before succeeding. don't export to a filename that you used or imported while editing. When you open a .wav file, use it in editing, and export, audacity needs the original file to produce the end result. So overwriting the original while audacity still needs it, does not work. exporting one stereo audio track will convert it to mono for some reason. You'll be left with sound on the left channel, and silence on the right channel. When you split the stereo audio track into two mono audio tracks, you can export the result as a stereo file (.wav, .mp3, whatever you choose). The rest of the operation was rather simple, and I now have an enhanced digital version of my late grandfather singing, recorded more than 30 years ago !

a website pretending to be a search engine, uses your input to fabricate results. Luckily, the ones that I've seen aren't doing much to hide this practice, and searching for "made_up_hokey_pokey" or another randomly invented string will make it very clear. Example of the day: torrentreactor.to. Every search result under "recently added" is being faked especially for you.

the minimal unit used for pricing is actually larger than what you use 99% of the time. Example of the week: Proximus mobile internet invoiced per 100KB, and my invoice says: 70 connections this month, 7 MB used.

...the new packaging of a product is physically larger, but contains less. Check the price, and I bet you it hasn't gone down. Example of the week: new Duyvis crac a nut packs are 175g, old packs were 200g, but were physically smaller.

Received my paper copy of The Definitive Guide to CentOS today. As the technical reviewer of this book, I've seen the content during the writing phase, but seeing it in print is still cool !

Creating a software RAID1 setup with one 2TB and two 1TB drives isn't difficult, but in this case the setup was running in production as a RAID1 over two 1TB drives. Small sda1 and sdb1 partitions form /dev/md0 and are formatted ext3 for /boot. Large sda2 and sdb2 partitions form /dev/md1 and are an LVM PV, the rest of the system is formatted in logical volumes. Target situation: to have /boot on all three disks, and the LVM volume group as large as possible, redundant on mirrorred diskspace. The procedure is strict, but straightforward: partition the new disk in 3 partitions (small - large - large) make md0 3-legged with sdc1 as the third device make md1 3-legged with sdc2 as the third device wait for the sync (watch /proc/mdstat) remove sdb2 from md1 and make it 2-legged again (you'll have to --fail sdb2 before you can --remove it). make a new md2 device for sdb2 and sdc3, make it a PV and add it to your volume group. The only utilities needed are cfdisk, mdadm, and lvm (for ...

Now that 's innovation: automatically detecting the license plate of cars entering a car park / gas station from a highway, and holding them back for a minute if they try to enter the highway too fast. People trying to cut traffic jams (and thereby making them worse, because merging traffic at unadapted speeds creates and worsens traffic jams) get penalized. Can we get this on the E40 in Heverlee, please ? Pretty please ?

A software-implemented iSCSI initiator (in VMware ESX) on one side, the iSCSI Enterprise Target (IET on CentOS5) software on the other side, and a separate storage VLAN to connect the two. Works great in my test lab. But then I wanted to add multipathing to the setup. Plan was: create a second VLAN, and give both the IET server and the iSCSI client a new interface in that VLAN, using a new IP subnet. That gives the client two ways to reach the server, thereby introducing multipathing ! Unfortunately, IET spews error messages at high speed when doing this: I got " kernel: iscsi_trgt: Abort Task (01) issued on tid:1 lun:0 by sid:26459747326427648 (Unknown Task) " about 4000 times per second. Restarted everything without the second storage VLAN and without the new interfaces, and now all is well again.

It's been a while since I made time to try something new. This week, I finally took something off of the "need to try this" list: link aggregation . I've had a gigabit Ethernet switch with link aggregation for about a year now, and my main Linux box has 3 gigE NICs, but I was still only using one. Time for change. Google found me some good documentation for channel bonding on CentOS5. Manually editing the ifcfg-eth{0,1,2}, ifcfg-bond0, and modprobe.conf is all that's required. That worked, but the default bonding setting is "balance-rr", the simplest loadbalancing algorithm. What I wanted to use was full IEEE 802.3ad link aggregation, mode 4 of the bonding module. During testing, I got fooled into believing that "service network restart" unloaded and reloaded the bonding module. It doesn't, I should have tested using "service network stop; rmmod bonding; service network start" from the start. Learned my lesson, configured the swit...

Fabian (of RPMforge and CentOS fame) wrote a blogpost about using pv (pipe viewer) to watch a datastream through a pipe, such as a dd-over-ssh copy from one machine to another. I knew pv existed, but I've often used dd on systems where pv wasn't available (e.g. rescue CD), or used dd without sending data through pipes. Even in these situations, you can keep an eye on dd's progress: open another terminal window, then run # while sleep 10; do killall -USR1 dd; done and switch back to the shell where dd is running. It will show its status every 10 seconds. Note, your version of dd needs to support this. The dd from RHEL3 doesn't (version 4), but the dd from RHEL5 does (version 5) !

Today at VMworld Europe 2009, VMware demoed a preview of their vCenter management server running on Linux in a virtual machine, pre-packaged in virtual appliance format. Aside from that fact that this is a great idea with all the advantages that virtual appliances have to offer, and that a lot of people have been waiting for, it was a really pleasant surprise to me that they chose to build it on top of CentOS 5 ! It'll take months before a final version will become available for download to VMware vCenter customers, but it's going to happen, that's for sure. Kudos to the CentOS team !

I used VirtualCenter (vCenter) to put an ESX in standby mode. I restarted VirtualCenter for Windows patches, and now VirtualCenter says the ESX host is "not responding", and doesn't give me the option to wake it from standby mode. Fortunately, the solution is right there under your fingertips: do manually what VirtualCenter would have done for you: Log on to another ESX on the same network, get the MAC address(es) of the sleeping ESX host, and run: # ether-wake -i vswif0 aa:bb:cc:dd:ee:ff You could optionally repeat this for every interface you know of the sleeping ESX. Remember to use the real MAC addresses of the real NICs, not the virtual MAC addresses of service console ports, vmkernel ports, or virtual machines. After a couple of minutes, VirtualCenter will notice that the host is responding, and will connect it again. If you wish, you can put it in standby mode again using the VI-Client. If you don't know the MAC address of the host in standby mode, you would ne...

Any sysadmin who runs postgresql (or sqlite) knows he should vacuum once in a while. You didn't ? Probably because nowadays, there's an autovacuum function. But what about the Real World (you know, the big room with the blue ceiling) ? After years of manually vacuuming the floors, I now got myself an autovacuumer to take care of my office space: the iRobot Roomba 560. This robot can drive around on the floor and cleans it. After having cleaned the room, it automatically returns to the dock where it recharges the battery. Aside from the "cool" factor, it really does a good job. I've seen comments from people who claim that a vacuum cleaner can't run on batteries for an hour, even proving their point with calculations, but don't be fooled: the Roomba is not a vacuum cleaner like the one you already own. First, doesn't suck dirt through a 3 meter tube. Second, it isn't even a vacuum cleaner in the strictest sense: it uses brushes to pick up most of th...