the birdhouse in my soul

Too much redundancy will kill you

2011-10-18T14:08:00.000+02:00

A customer asked me to verify their vSphere implementation. Everything looked perfectly redundant, in the traditional elegant way: cross over between layers to avoid single points of failure. I had to break the bad news: too much redundancy can mean NO redundancy.
In this case: host has 4 network interfaces (2x dual port card). VM's connect to a vSwitch, which has redundancy over vmnic0 and vmnic2 (using 1 port of each card). Another vSwitch for the storage traffic, same level of redundancy, using vmnic1 and vmnic3. Looking good.
Then the physical level. 4 host interfaces, 2 interconnected network switches. The traditional |X| design connects the two interfaces of every card to different switches. Looking good.

But looking at both configurations together, you'll see that every vSwitch gets connected to one physical switch. The sum of two crossed redundancy configurations equals no redundancy at all.
Enabling CDP or LLDP can help you identify this problem, as you can identify on every interface which physical switch it connects to. In this case the CDP physical switch identifier was the same on vmnic0 and vmnic2, and again the same on vmnic1 and vmnic3.
I advised changing the cabling to four straight || || connections, vmnic0 and vmnic1 to the left switch and vmnic2 and vmnic3 to the right switch. That re-introduces the redundancy they thought they had.

vCenter Appliance and underscores in hostnames

2011-09-12T21:44:00.000+02:00

Found out the hard way: don't use underscores in hostnames. It's not allowed by DNS, and it breaks things. In this case: joining vCenter Server Appliance (VCSA) in an Active Directory doesn't work if the hostname of the appliance contains an underscore (_). It also doesn't work if the hostname is "localhost".
If your appliance uses DHCP, the appliance gets its hostname through reverse DNS. So in that case, it _is_ a freaking DNS problem.

vSphere5 nested virtualization as seen in /proc/cpuinfo

2011-09-06T22:33:00.001+02:00

I won't blog about the whole vhv.allow="true" procedure here, that's been covered elsewhere. But what does nested virtualization change in a VM ? Well, the CPU features that are exposed change:
A regular 64-bit Linux VM sees

# grep flags /proc/cpuinfo
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc up arch_perfmon pebs bts rep_good xtopology tsc_reliable nonstop_tsc aperfmperf unfair_spinlock pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt aes xsave avx hypervisor lahf_lm ida arat

A 64-bit VM with nested virtualization enabled sees

# grep flags /proc/cpuinfo
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc up arch_perfmon pebs bts rep_good xtopology tsc_reliable nonstop_tsc aperfmperf unfair_spinlock pni pclmulqdq vmx ssse3 cx16 sse4_1 sse4_2 popcnt aes xsave avx hypervisor lahf_lm ida arat tpr_shadow ept vpid

So the vmx, tpr_shadow, ept and vpid features are hidden for a normal VM, and are exposed when you enable nested virtualization.
N.B. These /proc/cpuinfo examples were created on vSphere5 running on a Sandy Bridge E31270 CPU. Other CPU generations will show a different flag set, but the four added features should be the same.

SSH cipher speed

2011-08-19T10:56:00.001+02:00

When setting up backups over SSH (e.g. rsnapshot with rsync over SSH), it's important to know that the default SSH cipher isn't necessarily the fastest one. In this case, the CPU-based encryption is the performance bottleneck, and making it faster means getting faster backups.
A test (copying a 440 MB file between a fast Xeon CPU (fast=no bottleneck there) and an Atom based NAS) shows that the arcfour family of ciphers are clearly the fastest in this setup:

cipher	real time	user time	bandwidth
arcfour	0m9.639s	0m7.423s	45.7 MB/s
arcfour128	0m9.751s	0m7.483s	45.1 MB/s
arcfour256	0m9.856s	0m7.764s	44.7 MB/s
blowfish-cbc	0m13.093s	0m10.909s	33.6 MB/s
aes128-cbc	0m22.565s	0m20.129s	19.5 MB/s
aes128-ctr	0m25.400s	0m22.951s	17.3 MB/s
aes192-ctr	0m28.047s	0m25.771s	15.7 MB/s
3des-cbc	0m51.067s	0m48.018s	8.6 MB/s

The default configuration of openssh uses aes128-ctr, so changing the cipher to arcfour gets me a 2.5-fold increase in bandwidth here ! Use the "Ciphers" keyword in .ssh/config or the "-c" command line parameter to change the order of preference of the available ciphers. YMMV.

As a reference (cfr. deinoscloud's comment), I ran "nc -l -p 3333" on the Atom side, and ran "cat file | nc atom 3333" on the Xeon:

cipher	real time	user time	bandwidth
cleartext	0m4.135s	0m0.311s	106.5 MB/s

. This shows that in the cleartext case, the CPU (user) time is not the bottleneck, and we're very close to using the full 1Gbps bandwidth.

Dell's R210-II as vSphere home lab server

2011-08-15T20:19:00.000+02:00

My VI3 and vSphere4 home lab consisted of whitebox PCs. For VI3 I used MSI based nonames, for vSphere4 I used Shuttle SX58j3. For the new vSphere5 generation, I wanted some real server hardware. Because of shallow depth requirements, the choice of rackmount servers was limited. I picked the Dell Poweredge R210II instead of the sx58j3 because
- on the vSphere HCL (the sx58j3's won't boot vSphere5 RC !)
- Sandy Bridge low TDP CPUs available (I got the E3-1270)
- onboard dual BCM5716 nics support iSCSI offload (aka "dependent HW iSCSI")
- IPMI built-in (not tested yet)
- dense: 1U (the sx58j3 is about 4 units, but can fit 2 in 19")
- one free PCIe slot (The sx58j3 has 2 slots, but needs a VGA card)
- not incredibly expensive (up to 16GB RAM)
Downsides:
- only one free PCIe slot (max GbE nics needs expensive quadport card)
- incredibly expensive (with 32GB RAM it's 3x the price of a 16GB config)
- can't buy without at least one disk. I'll be running from USB sticks.

HTTPS SSL stops working because of old libraries

2011-08-06T17:05:00.000+02:00

At a customer, a Linux workstation suddenly refused to open HTTPS sites. Verified recent package versions of both browser (konqueror) and libraries (kde, openssl), everything looked good, but it didn't work. This blogpost serves as documentation for the fact that checking new software isn't enough, because in this case removing old openssl compatibility libraries solved the problem. The kio_http helper is not linked with openssl directly, and for some reason it must have tried to open one of the old openssl versions that were also installed. After erasing all versions between 0.9.5a and 0.9.6b, keeping the current 0.9.8e, konqueror had no problems opening https sites anymore.

Home lab switch

2011-06-22T19:19:00.000+02:00

My home lab got upgraded with a new gigabit switch recently. Main improvement I wanted over the old Linksys SLM2024 I had: Cisco Discovery Protocol.
Based on that requirement and the budget, I selected the Cisco SG300-28 Small Business managed switch. The web interface is clearly improved compared to the SLM2024, and CDP is a real treat. Both vSphere ESXi and the cdpr utility under Linux decode the CDP information nicely. CDP is a great help to find errors in patch cable arrangement !

Logitech diNovo Mini keyboard lacks F-keys

2011-04-01T19:01:00.000+02:00

I thought the Logitech diNovo Mini keyboard would be a perfect keyboard to keep in my basement rack for occasional maintenance activities on my Linux and vSphere servers. Turns out the diNovo Mini lacks F keys. Not even Fn-[number] will send the correct keycode. What a disappointment. The larger (but still small) diNovo Edge has function keys, but is far less suited to be left in a dusty environment like a basement rack.
Does anyone else know of a better solution ?

Weird vmnic numbering

2011-03-30T11:31:00.002+02:00

After installing new Intel quad port ethernet cards in vSphere ESXi machines, I had to figure out which physical port matched to which vmnic number. Strange though it may sound, the mapping turned out to be (top to bottom as seen on the back of the card).

A: vmnic2
B: vmnic3
C: vmnic0
D: vmnic1

However, the PCI layout of most quad port cards makes this easier to understand: a quad port card is implemented as two dual port cards behind a PCI bridge chip. While enumerating the PCI bus, the VMkernel can find one bus first, enumerate the devices on it, then find the second bus, and enumerate the devices there.
In this case, the bottom bus was found first, and vmnic's on it were counted top to bottom (vmnic0 and vmnic1). Then the top bus was found, and again vmnic's on it were counted top to bottom (vmnic2 and vmnic3).

When marketing and technical information meet: Hyper-V

2011-02-28T22:02:00.004+01:00

While reading an article about Hyper-V per-VM CPU settings, I saw this in the FAQ:

[BEGIN QUOTE]

Why do you use percentage for the limit and reserve – and not MHz / GHz?

Many people find it easier to think in MHz / GHz rather than percentage of a physical computer. They also argue that using a percentage means that as you move a virtual machine from computer to computer you may get different amounts of resource depending on the underlying capability.

This is something that has been discussed extensively on the Hyper-V team, and while I do believe there is some merit in this approach, there are a number of reasons why we chose to use a percentage instead. Two key ones are:

Predictable mobility

If all your virtual machines have a reserve of 10% – you know that you can run 10 of them on any of your servers. The same would not be true if they all had a reserve of 250Mhz. Given how important virtual machine mobility is to our users – we believe that this is something that needs to be easy to manage.
Not all MHz are the same

1GHz on a Pentium IV is much slower than 1GHz on a Core i7. Furthermore – newer processors tend to be more efficient at virtualization than older processors, so the difference between the “bang for buck” that you get out of each MHz varies greatly between processor types. This means that in reality – defining a reserve or limit in MHz / GHz does not really give you a great performance guarantee anyway.

[END QUOTE]

Even though this seems to be a list of technical arguments, the claims made are non-sensical:

"we use a relative percentage instead of a fixed unit because we want you to be sure you can run a certain number of guests on any CPU." What ?? Who says that my VMs will actually still run when they suddenly get only half of the power they needed, because they were moved to a CPU with half the horsepower ? A reserve is supposed to be a guarantee, a limit is supposed to be just that: a limit. Even the examples they give for using a reserve or a limit would fail. A misbehaving app that sucks CPU, will suddenly be allowed to use even more, just because it's now running on a faster CPU.
"Not all MHz are the same." That's not a very good reason to use percentages instead, is it. Are they claiming that every % _is_ the same ?

Dear Microsoft (and any other company reading this), please make your technical information technical, and correct. Do whatever you want with your marketing docs, but don't let the marketing seep into the technical documentation.

Every error is a DNS error.

2011-02-28T20:33:00.002+01:00

Newly installed RHEL5 machine in an existing network. Users opening firefox on the machine got an error "The bookmarks and history system will not be functional". The googlesphere suggested renaming places.sqlite and such, but that didn't help. Things began to clear up when I found errors on the NFS server that exports the home directory: "lockd: failed to monitor newmachine.companydomain". I checked the nfslock service, but it was running fine. Configuration files for NFS and autofs were identical to other machines that didn't show the problem. Then, like a bolt of lightning, it hit me: I had forgotten to create a reverse DNS entry for the new machines IP. Forward DNS was OK, but reverse wasn't. That caused the NFS lock error, and that caused the firefox error... The old saying is confirmed once more: every error is a DNS error.

Link aggregation and VLANs on QNAP with firmware 3.4.0

2011-02-19T16:45:00.003+01:00

The new QNAP firmware (3.4.0) supports 802.1q VLAN tagging, but you can't create multiple interfaces in different VLANs on the same physical interface through the webinterface.

In the case of link aggregation (LACP 802.3ad for example), that means only 1 VLAN and 1 IP address can be used.

Fortunately, QNAP allows full access to the underlying Linux system. Adding a VLAN interface goes like this (the example uses VLAN 234)

# /usr/local/bin/vconfig add bond0 234

# ifconfig bond0.234 192.168.2.30 broadcast 192.168.2.255 netmask 255.255.255.0

of course, this change is not permanent, a reboot will not automatically start this interface. I'll blog about making it permanent later.

software RAID on old vs. new CPUs

2011-02-19T12:59:00.001+01:00

The Linux kernel has several software RAID algorithms, and selects the one that is fastest on your CPU. Isn't that always the same algorithm then ? No, definitely not. Newer CPUs have additional instructions that help speed things up. And it's not just clock speed that matters, memory bandwidth plays an important role too.

On an old Pentium II Xeon 450 MHz, raid5 uses p5_mmx, and raid6 uses mmxx2. Software raid6 calculations are 72% slower than raid5.
On a Pentium IV Xeon 1.5 GHz, raid5 using pIII_sse, and raid6 uses sse2x2. Software raid6 calculations are 12% slower than raid5.
On an AMD Athlon XP2000+ (1.6 GHz), raid5 uses pIII_sse, raid6 uses sse1x2. Software raid6 calculations are 42% faster than raid5.

On 64-bit systems, no relevant instructions are different between generations so far:

On a AMD Athlon64 XP3400 (2.4 GHz), raid5 uses generic_sse, raid6 uses sse2x4 (raid6 44% slower than raid5).
On a Xeon 5160 3GHz, raid5 uses generic_sse, raid6 uses sse2x4 (raid6 15% slower than raid5).
Same algorithms on a Xeon X5450 3GHz (raid6 20% slower than raid5).
Same algorithms on a Xeon E5430 2.66GH (raid6 18% slower than raid5).
Same algorithms again on a Xeon X5650 2.66GHz (raid6 15% slower than raid5).

RHEL6 comes prepared for vSphere

2010-11-13T22:23:00.002+01:00

A fresh install of RHEL6 contains several vSphere-ready components:
the standard kernel package contains kernel modules for the optimized VMware virtual hardware (network, storage, and memory balloon driver).

vmxnet3
vmw_pvscsi
vmware_balloon

and just like in earlier RHEL releases, there's drivers for the VMware graphics card and the mouse driver:

xorg-x11-drv-vmware
xorg-x11-drv-vmmouse

Especially the built-in network and storage drivers will make life easier for RHEL admins in vSphere environments.
That's great news ofcourse, but I'd like to stress that this is not equivalent to a full VMware Tools install, which would include extras such as shutdown/reboot/freeze/resume scripts, IP address display in the vSphere client, etc.

Enterprise Linux and state-of-the-art hardware: a difficult marriage

2010-11-07T11:29:00.002+01:00

A customer bought a Dell Precision T7500 recently. Beautiful machine, awesome power, running CentOS5. The videocard in the machine was the first I ever saw with nothing but DisplayPort connectors: no old-style VGA, no DVI, no HDMI. At first, a 1680x1050 screen was connected using a DisplayPort-to-DVI connector, and that worked flawlessly.
But in came a new beast: a 30-inch 2560x1600 screen. The standard DVI cable can't handle this resolution, and neither can the DisplayPort-to-DVI connector, so even a dual-link DVI cable doesn't solve this issue. And I'm sure you'll agree: running such a beauty at 1920x1200 seems a waste.
I tried connecting the monitor directly using a DisplayPort cable, which can easily handle the maximum resolution, but the EL5/CentOS5 xorg-x11-drv-ati driver is too old: version 6.6.3 (with some backports from 6.12.2) doesn't detect DisplayPort-connected monitors correctly. Compiling a newer driver isn't easy, because the newer ones require Xorg 1.2 or 1.3, not the 1.1 version in EL5.
So until EL6 comes out (later this year, we expect), I'll need another solution. A three year old NVidia card I had lying around still had a real dual-link capable DVI output.
Replace the card, boot the machine, /etc/X11/xorg.conf gets updated to use the "nv" driver, and the GDM login screen comes up in ... 1920x1200. RTFM translated to "man nv" in this case. After reading the manual page, add
Option "AllowDualLinkModes" "true"
to the Device section of /etc/X11/xorg.conf, restart gdm (init 3; init 5), and voila.

Which ESX version am I running on ?

2010-10-16T10:46:00.003+02:00

(An update of an older post: now with vSphere 4.1 info. Further updated in 2011 with vSphere 5 info.)
Your Linux runs on a VMware VM, but which ESX version is it ? You can see for yourself (as already explained in an earlier post on this blog). Run "dmidecode" and look at lines 10, 11 and 12. The list has been updated with current info:
ESX 2.5 - BIOS Release Date: 04/21/2004 - Address 0xE8480 - Size 97152 bytes
ESX 3.0 - BIOS Release Date: 04/17/2006 - Address 0xE7C70 - Size 99216 bytes
ESX 3.5 - BIOS Release Date: 01/30/2008 - Address 0xE7910 - Size 100080 bytes
ESX 4 - BIOS Release Date: 08/15/2008 - Address 0xEA6C0 - Size 88384 bytes
ESX 4U1 - BIOS Release Date: 09/22/2009 - Address 0xEA550 - Size 88752 bytes
ESX 4.1 - BIOS Release Date: 10/13/2009 - Address 0xEA2E0 - Size 89376 bytes
ESX 5 - BIOS Release Date: 01/07/2011 - Address 0xE72C0 - Size 101696 bytes

USB-over-IP goes mainstream

2010-10-15T18:34:00.003+02:00

I've been into virtualization for a long, long time, so I was familiar with the USB-over-IP concept. It has always been the preferred way to bring USB into a virtual machine, because it doesn't limit the flexibility of virtualization: live migration (vMotion), failover (HA), fault tolerance, ... can all handle USB-over-IP.
But I was still surprised to find a cheap USB-over-IP device in a local computer store: the Belkin Network USB Hub, NUH for short. 100 USD list price, became EUR 90 retail price here in Belgium. Not the greatest deal around, but no reason to feel grumpy.
Now let's see what this baby can do: the test setup consists of the Belkin NUH and two clients: a Windows Vista 32-bit laptop connected over WiFI and a Windows 2008 R2 64-bit VM on VMware vSphere connected over wired GigE. Then I gathered a diverse set of USB devices: USB memory sticks, USB hard drives, a USB smartcard reader, and a USB CD/DVD writer.
The NUH gets a DHCP address by default (can be changed to a fixed IP). The Belkin software on each client detects the NUH on the LAN, shows you which USB devices are plugged in, and which system name is using each USB device.
Claiming a device is easy, and after installing a suitable driver, the device is ready to use. I didn't encounter problems using any device I tested !
So all in all, it works very well, and as easy as can be. However, as a virtualization user, I've got to consider home use as well. And I have to be honest: this device is probably not for production use. Why ? Let's look at both sides of the medal:
Pro's:

5 USB ports can each be used by a different system
Easy setup
Wide compatibility. Every USB device that I tested worked. Even a webcam worked, even though Belkin says they don't guarantee the functionality of webcams and some other devices
Relatively cheap. I've seen solutions 3 to 5 times more expensive, including some with less functionality.

Cons:

Windows only. That's a pity. The NUH runs an embedded OS (doesn't seem to be Linux however), and the protocol is probably Belkin-specific (I guess?).
Security aimed at home use: no passwords, no authentication. Every computer on the network can connect, see which USB devices are there, see who's using them, and connect to unused devices. True, the NUH can firewall (allow or deny) a couple of IP ranges, but anyone who can reprogram his own IP address on the LAN, can circumvent that. Using one NUH per OS and allowing just that one single IP address is the most secure option, but not the cheapest, nor the most manageable one.
Not that fast. I saw a sustained 3MBps from a client to the NUH, which is not the peak performance a USB disk can do.

QNAP to support VLAN tagging in firmware 3.4

2010-10-13T00:36:00.002+02:00

Good news for those of us with QNAP units in our environments. On the VMworld partner exhibition floor, my sources confirmed that VLAN tagging support has been implemented, and will be released in firmware 3.4. QNAP has release 3.3.3 recently, and it's unknown when 3.4 will be released.

ESX home lab upgrade

2010-10-11T18:20:00.004+02:00

My original home lab, three years old now, included two PC's running virtualization software, with Intel core 2 quad CPU's and 8 GB RAM each. While browsing for a replacement, I found Didier's Shuttle SX58J3 review. I researched some alternatives, but decided to go for the SX58J3 as well, fitted with 12 GB RAM and i7-970 CPUs, hexacores with hyperthreading.

shallow: depth is critical in my telco rack.
reduced height: I might get 4 shuttles in the same space as 2 minitowers earlier. They're actually slightly wider, but not by much.
12 GB per server is 50% more than the old lab (16 GB is supported but slower)
the Intel core i7-970 CPU gives 100% more MHz than the old Q6600. They are much more expensive, truth be told. But being Westmere generation, they support vSphere4.1 features like DirectPath I/O and Fault Tolerance.
I can re-use the PCIe dual gigabit adapters I had in the old lab.

One real problem so far: in BIOS v100, I couldn't change any setting without rendering the machine unbootable (hanging on "checking NVRAM"). With default settings, I could install Vista and update to BIOS v102. Booting problems were then solved, and the machines now boot ESXi 4.1 from a USB stick.
A minor issue I still have to resolve: putting ESXi in standby mode works, reviving them via Wake-On-LAN doesn't (yet).

Comparing bluetooth carkits

2010-08-25T18:54:00.004+02:00

Bluetooth carkits come in three main categories: those with bluetooth HSP, those with bluetooth HFP, and ones that do bluetooth SAP. Some implement multiple of these profiles and allow you to choose.

HSP (head set profile) is the simplest profile: your carkit is basically a speaker-microphone combo, and will play the received audio and send your voice to the telephone. The telephone connects to the cellular network and handles the call. HSP is commonly used in bluetooth earpieces.
HFP (hands free profile) is the most common protocol in carkits with a display: your phone still handles the call, and the carkit provides the two-way audio function just like HSP. But the carkit also has basic control over the phone: it can access the received/dialled/missed call lists. It can also instruct the phone to dial a number, to accept or to reject an incoming call. Most carkits also access the phonebook.
SAP (SIM access profile, also known as rSAP or SIM) is the most complex of the three: this carkit contains a cell phone ! The mobile phone will disconnect from the network, and lend its SIM card to the carkit phone. Calling is now done by the carkit phone. The carkit will typically also sync the phonebook of the phone via another bluetooth profile.

this is my list of pros and cons for all:

HSP pro: your phone can do everything it wants, and you speak to the carkit the way you'd speak into your phone. You still have 3G, and if your phone does smart tricks like voice commands, that probably still works over the carkit. You can seamlessly disconnect the bluetooth kit during a call and continue the call on your phone.
HSP con: your phone is the only place to control calls, see caller ID, etc. Also, your phone emits radiation inside your car.

HFP pro: you can still use your phone to control calls, but also the carkit display and buttons. On your phone, you'll still have 3G. In my experience, things like voice commands for your phone won't work anymore, but maybe your carkit has similar functionality. As with HSP, you can seamlessly disconnect/connect the carkit during a call.
HFP con: your phone emits radiation inside your car, although with a suitable cradle, your mobile phone may benefit from an external antenna. Depending on phone and carkit, you may not hear your personal ringtone anymore.

SAP pro: no more phone radiation inside your car! The carkit has an external antenna, which means better reception. Battery usage on your mobile phone is minimal.
SAP con: you can't seamlessly switch from carkit to mobile phone. Getting in your car during a call means ending the call and redialing from your carkit phone. Your phone can't do 3G anymore while connected to the carkit. Not all phones support this profile (most notably the iPhone doesn't). You don't hear your fancy personal ringtone anymore, because it's now you carkit phone that's ringing.

Am I running in a VMware virtual machine ?

2010-08-03T11:09:00.003+02:00

That's an easy question, and the answer isn't too difficult either:

Query the MAC address of the local network card. If it starts with "00:50:56", that indicates that it's a VMware VM.
List devices on the PCI bus. If there's devices with vendor ID 15ad, you can be sure that this is a VMware VM.
Look at the BIOS information (DMI). If you see Manufacturer "VMware, Inc", and the serial number starts with "VMware", and the Product Name is "VMware Virtual Platform", that's again very clear.
In a typical VMware VM, you should find the VMware Tools running: the vmmemctl driver, maybe the vmxnet network card driver maybe, the vmware-guestd or VMwareUser or VMwareTray processes, ...

And here's a practical list of tools you'd use to run these checks:

use "/sbin/ifconfig eth0" on Linux, "ipconfig /all" on Windows. You can do this as a normal user.
use "/sbin/lspci" on Linux. Also possible as a normal user.
use "dmidecode" on Linux, or the third-party tool CrystalDMI on Windows. This is only possible with administrative privileges.
use "ps auxfw" and "ls /proc/vmmemctl" on Linux, Task Manager on Windows. Can be done as a normal user.

P.S. I posted earlier how you can see which version of ESX/ESXi you're running on. This technique diggs further in the dmidecode/CrystalDMI information.

DIY Optical Parking Sensor upgrade

2010-07-23T23:16:00.005+02:00

My 2008 Volkswagen Touran came with Parking Distance Control, beeping to indicate the remaining distance to objects when the car is in reverse gear. The control unit that I had was of an old type, and I had the impression it wasn't functioning 100%. So I got a new module from Kufatec, to replace the old one.
Kufatec explains that you need to run a wire from the back of the vehicle to the front, but as it turns out this is not necessary when the old module is already at address 10 on the CANBUS network. If the old module is at address 76, then you do need the new wiring. I already ran the wire to the front of the vehicle and connected it before realizing that I didn't need it. Result: countless electronics error messages in the on-board diagnostics. It could have been a simple drop-in replacement, if the documentation would have explained this correctly. Clearing all the error messages took quire some time, but was succesful in the end.
The new module works well now, and as a bonus, it gives me visual status of the four individual parking sensors (OPS) in addition to the beeps (regular PDC). This Youtube clip shows what it looks like after the upgrade.

death grip

2010-07-19T18:58:00.003+02:00

Yes, my phone has a "death grip". No, it's not a v4 iPhone, it's a Nokia E71. But the comforting thing is, Nokia documented it in the manual all along (page 16/17). That indicates that they knew this while designing the phone. That figures, because covering/connecting both antenna areas would require a very awkward grip. Whereas the reaction Apple displayed recently has a distinct "Oh f*ck, we forgot about that" feel to it.

VMware tools on RHEL/CentOS: the easy way

2010-07-09T17:23:00.002+02:00

VMware pre-compiles the VMware tools for selected OS kernels. The stock RHEL kernels are included, but not the intermediate updates. If you can live with that, you can simply add the VMware tools YUM repository:
# cd /etc/yum.repos.d/
# wget http://bert.debruijn.be/linux-stuff/vmwaretools.repo
and download the VMware RPM signing key
# cd /etc/pki/rpm-gpg
# wget http://packages.vmware.com/tools/VMWARE-PACKAGING-GPG-KEY.pub

Then install the tools packages:
# yum install vmware-tools-nox
or
# yum install vmware-tools

vCenter will report the tools version as "Unmanaged" rather than "OK", but you have heartbeat (so HA and alarms can detect guest OS crashes), balloon driver, etc.

DIY carkit replacement

2010-07-03T15:01:00.008+02:00

The factory-installed UHV carkit[0] in my Volkswagen Touran worked well with a normal cellphone-holder, but in combination with the Bluetooth Touch Adapter[1], it became a very unreliable combination. I've read many succes stories with the Touch Adapter, but the flaky communication between the TA and the UHV was a nightmare sometimes.

To fix that, I ordered a replacement carkit from Kufatec[2]. The module replaces the simple factory UHV with a newer one (also original from VW). The new module does not only bluetooth sound, but also address book, call lists, etc. It doesn't offer cellphone specific holders for power and antenna connections.

Installation was quite easy, as there are many howto's describing VW Low-to-Premium modding. In the Touran (1T0), the UHV is under the passenger seat, and requires removal of the lowest drawer. A T15 Torx screwdriver does the trick. Then you can open the floor cover, get the original module out, and the new module in. Replace the drawer, and that's it. You'll just need to program the module, so that it knows how to talk to the rest of the car. You'll get phone menu's on the MFA+ display in front of the driver, and on the RNS/RCD navigation system in the central console. Any VW workshop with VAS or VCDS can do that for you.

[0] Factory installed unit: 3C0 035 729G. Only does sound over BT. No screen output.
[1] VW Bluetooth Touch Adapter: 3C0 051 435 TA. Does address book, call list, and sound with a phone, and passes the sound through to the UHV unit [0]. Has its own little touch screen.
[2] VW replacement part from Kufatec: 1Z0 035 729B. This does address book, call list and sound natively, and displays it on the MFA+ and RNS510 screens.

P.S. I didn't remove the passenger seat for this mod. If you don't already have a VW carkit, or you're changing to the VW Premium car kit (the one that does rSAP, and has its own antenna), that's probably a different story.

QNAP 459U-RP: first impressions

2010-06-26T11:53:00.004+02:00

I recently added a dedicated storage box to my lab environment. After a week these are my first impressions:

Pro:

Small. I needed a shallow rack-mountable device, and this one is perfect, only about 50cm deep.
Fast. I knew that a NAS device like this wouldn't be the speed king of the storage world, but I'm pleasantly surprised.
Open software. It runs Linux, you get SSH access to your device, and you can add packages if you want. Evidently, I added tools like dstat to keep an eye on things.
Compatibility. Used it from Windows Vista, VMware vSphere, and Linux (CentOS and others). No problems at all.
Cheap. I could have gone for an EMC Symmetrix instead, but decided against it :-)

Con:

Fragile. Just a bit. The SATA ports of the disk drives slide directly in sata plugs inside the device. I hope I won't have to re-plug drives all too often.
Not all operations happen online. QNAP advertises RAID1 to RAID5 migration, and it does that, by using one of the mirrored drives and the new drive to build the RAID5 array, then moving the data from the broken mirror to the incomplete RAID5 array. You'll be seeing a different filesystem after the data move.
Robustness. One of the four drives I bought with the device (WDC-something-FYPS SATA disks) was DOA. The QNAP refused to use the slot that was occupied by the faulty drive, until after a power cycle.
Bugs. All software has bugs, and this version of the firmware has a bug in the iSCSI configuration wizard. I'll put my VMs on NFS for now...
No VLAN tagging support. I've got a separate IP storage VLAN, and to see the QNAP on both the regular LAN and the IP storage VLAN, I need to use one interface per VLAN. Why not support VLAN trunking ? The 8021q driver is already in the QNAP software, but the vconfig utility isn't, and the web interface would need to support it as well.

Scanning a host reconfigures the cluster. What and why ?

2010-06-12T13:03:00.002+02:00

On vSphere4, when you scan a host (ESX or ESXi) for updates, you'll see a task "reconfigure cluster" appear automatically. Ever wondered what that task does ? The "tasks & events" tab doesn't give you any clues, but if you happen to be looking at the "DRS" pane while the task is running, you'll see it: scanning a host in a cluster, automatically disables DPM for the duration of the scan. My guess is that this is to prevent DPM from putting a host in standby mode while it is being scanned.

Buy VMware, get SuSE

2010-06-09T23:22:00.005+02:00

Anyone following VMware's enterprise offerings will know that VMware has been "in bed" with Red Hat for a long time. The "service console" of the ESX product has been Red Hat Linux and Red Hat Enterprise Linux based since the beginning, and many of the virtual appliance-based products (VMware Data Recovery e.g.) are RHEL or CentOS based.
But now VMware comes with a partnership announcement with Novell/SuSE, not Red Hat ! Anyone buying vSphere will get SLES for free, including patches ! Note, the offer isn't fully up just yet, they forecast a GA launch in 3Q2010. But anyone buying vSphere starting today (june 9th) will be in on the offer when it launches. Beware, "free" doesn't include support, from what I read in the press release. But SLES support will be available through VMware.
A lot of questions are waiting for an answer:

Will this deal impact the place RHEL and CentOS held in the VMware solution stack ?
What will be the response from Red Hat ? I'm sooo curious !
Has Novell got an exclusive deal with VMware, or is there room for a similar deal with Red Hat ?

Time will tell.

Which ESX version am I running on ?

2010-04-11T22:52:00.002+02:00

Your Linux runs on a VMware VM, but which ESX version is it ? You can see for yourself (as already explained in an earlier post on this blog). Run "dmidecode" and look at lines 10, 11 and 12. The list has been updated with current info:
ESX 2.5 - BIOS Release Date: 04/21/2004 - Address 0xE8480 - Size 97152 bytes
ESX 3.0 - BIOS Release Date: 04/17/2006 - Address 0xE7C70 - Size 99216 bytes
ESX 3.5 - BIOS Release Date: 01/30/2008 - Address 0xE7910 - Size 100080 bytes
ESX 4 - BIOS Release Date: 08/15/2008 - Address 0xEA6C0 - Size 88384 bytes
ESX 4U1 - BIOS Release Date: 09/22/2009 - Address 0xEA550 - Size 88752 bytes

Stating the obvious

2010-04-08T13:43:00.002+02:00

ICT website Computable runs a series of articles on Linux. The first article focuses on "hidden costs when migrating to Linux". In my experience, any project has hidden costs, and certainly so if preparation hasn't been done thoroughly, and with the necessary experience and know-how. There's plenty of experience available from various consultancy providers, use it !
PS don't forget to read the comments below the article as well.

Microsoft follows Red Hats lead: drops Itanium

2010-04-05T21:33:00.003+02:00

Red Hat announced in december 2009 that RHEL support for Intel Itanium was being dropped. On april 2nd, Microsoft announced that Windows 2008 R2 was going to be the last Windows version to support the Itanium platform. A chronicle of a death foretold...

Cisco SLM2024 and Firefox

2010-03-27T23:11:00.003+01:00

I don't know what got me to search for new firmware for my Linksys (now Cisco) SLM2024. But I did, and found version 2.0.0.8 (upgrade from the previous 1.0.1). Aside from the web interface, there's no obvious changes. It is now branded Cisco, rather than Linksys. But the most important change for me wasn't in the release notes: I can now manage my gigabit switch using Firefox ! (tested with Firefox 3.6.2 on Windows)

HZ divider effect on timer interrupt overhead

2010-03-05T22:31:00.004+01:00

Red Hat and related distro's (like CentOS) use 1000 timer interrupts per second, per CPU core or thread (this is called the "HZ value" inside the kernel). Because this causes a lot of extra work in case of virtualization, and caused (past tense since RHEL 5.4!) problems with timekeeping, the "divider" kernel parameter has been introduced. For example, by booting with "divider=10", the kernel uses 100 timer interrupts instead of 1000, and "divider=25" means 40 timer interrupts per second.
I did a little test today to see what difference that makes when running CentOS5.4 on vSphere. Tests were done with the current 2.6.18-164.11.1.el5 x86_64 kernel in a single vCPU VM. These are the results from my test environment:

standard settings: HZ=1000: 60 MHz cycles used when idle.
divider=2: HZ=500: 36 MHz cycles used when idle.
divider=5: HZ=200: 25 MHz cycles used when idle.
divider=10: HZ=100: 20 MHz cycles used when idle.
divider=25: HZ=40: around 10-15 MHz cycles used when idle (results slightly variable, also for higher dividers)
a nohz "tickless" kernel (I used 2.6.24.7-146.ay, not currently available in RHEL/CentOS by default) used 70 MHz cycles when idle in my test environment. This is a surprise, and I don't have a good explanation (yet) !

Note that 40 is the maximum divider setting. Using 41 will show "tick_divider: 41 is out of range" in dmesg, and the parameter will be ignored.
On your own machines, you can easily check:

the current divider setting by looking at the currently active kernel parameters: cat /proc/cmdline . No divider parameter means divider=1, the default.
the measured timer interrupt rate (close approximation of HZ value): dstat -i -I timer

Gaining 20-40 MHz of CPU power on your virtualization host might not seem important, but this is a per-guest win ! Do this on all your VMs, and you could be freeing several hundreds of MHz per host, and get more VMs, better consolidation ratio's, better performance, or lower power usage.

Summary: before RHEL 5.4, divider=10 was recommended for timer accuracy. This is no longer true, but as I've shown, it still helps lowering the timer interrupt overhead. Don't forget that the ideal divider setting depends on your application: thread wake-up delays can occur in high divider scenario's, and responsiveness could potentially suffer because of that.

Volkswagen UHV bluetooth touch adapter & its problems

2010-02-28T14:56:00.004+01:00

My Volkswagen car has the "universal cellphone preparation" UHV built-in. This is the main part of a car kit, but requires an additional adapter for connecting to a cellphone. At first, I was using an adapter for my good old Nokia 6310, even after I changed to the Nokia E71. Connecting was easy: pair the phone with the "VW UHV" bluetooth entity, and done. This has the phone connected to the car kit at all times, so even non-call-related functions use the car audio system (e.g. voice recognition).
But progress will have its way, no matter what happens. So in comes the "bluetooth touch adapter". Instead of a phone-specific adapter, this is a small touchscreen device that slots into the UHV dashboard mount. Connecting a phone is very different now:

the Bluetooth Touch Adapter connects to the "VW UHV" device via bluetooth
the phone connects to "Touch Adapter" device, also via bluetooth

The device doesn't allow step 2 if step 1 didn't succeed. Apparently this complex setup was necessary to allow the Bluetooth Touch Adapter to request phonebook, call list, and SMS information from the phone.

But it's not just good news: because the audio link from phone to car isn't direct, non-call audio functions don't work. The phone thinks it'll receive audio through BT, so doesn't use its own mic, but there's no BT audio feed because the BT Touch Adapter knows there's no call going on. And maybe worse: from time to time, I've experienced connection problems in step 1: the Bluetooth Touch Adapter fails to connect with VW UHV. With the limited UI of the device, this is almost impossible to troubleshoot. This seems to happen mostly after I've unlocked the car, and then closed it without ever turning on the ignition. On opening and starting the car minutes later, the connection failure happens.

the only remedy I've found is to turn on the ignition, "open" the VW UHV for new connections (press the phone button on the steering wheel twice), and keep doing that (every 5 to 10 seconds) until the unit connects. If it still fails, keep "opening" the VW UHV unit and use the "Settings", "Bluetooth", "Connect UHV" function on the BT Touch Adapter to retry. I'm hoping this will be fixed in a firmware update some day...

Preparing Linux for optimal thin provisioning and deduplication

2010-02-18T00:57:00.004+01:00

VMware and disk-array based thin provisioning can help you economize on disk space, just as disk-array based deduplication can. But how to take optimal advantage of those techniques ? Executive summary:

dd if=/dev/zero of=/tmp/tempfile.zeroes ; rm -f /tmp/tempfile.zeroes

First of all, both thin provisioning and deduplication work transparently. But you can make them work better with a little bit of work. There's three different types of blocks on your (virtual) disk that we need to distinguish: data blocks, old data blocks, and empty blocks.
Data blocks and empty blocks are what they are, and can't be influenced. Data blocks contain data for files that are on your filesystem. Empty blocks are empty, have never been written to with any real data, so their blank, contain only zeroes.
It's the "old data" blocks that we can improve: they contain data that used to be part of a file. The file got deleted, but the contents are still there.
Overwriting those blocks with zeroes will help a (re)conversion to thin provisioning later, and for dedup they are now empty blocks again, so perfect sharing possibilities.
Overwriting just the old data blocks is hard, but you can probably live with overwriting both empty and old data blocks. This will allocate all blocks (byebye thin provisioning, for now), but the contents will be all-zeroes: perfect thin re-thin-provisioning later, and perfect for dedup.
for each local filesystem mounted on your system, you can execute

dd if=/dev/zero of=/$mountpoint/tempfile.zeroes ; rm -f /$mountpoint/tempfile.zeroes

be careful, this will - very briefly - fill up your filesystem(s). If your application can't handle this, do it when that app isn't running. For those of you who use LVM, the unallocated PE's in every VG aren't cleared by this procedure, so for every Volume Group, find out name and available free space:

vgs -o vg_name,vg_free

then run (fill $vg_free and $vg_name with the data you just gathered)

lvcreate -n zerolv -L $vg_free $vg_name && dd if=/dev/zero of=/dev/$vgname/zerolv ; lvchange -a n /dev/$vgname/zerolv && lvremove /dev/$vgname/zerolv

DLNA with Linux and a Yamaha RX-V2065

2010-02-13T15:16:00.002+01:00

The Yamaha RX-V2065 is the first non-PC-based surround sound system I own. In addition to playing everything radio and hdmi-based, this receiver has an Ethernet connection. It can stream a long list of Internet radio stations, and it can find DLNA media servers on the local network to play various audio formats (no Ogg Vorbis though).
I've got my collection of some 400 CD's in mp3 format stored on my CentOS Linux fileserver, so obviously I wanted to access these files from the RX-V2065.
First attempt: Coherence 0.6.6.2 on the CentOS 5.4 fileserver. Because of missing dependencies, only the FSStore backend worked. Streaming worked, but no Track/Artist/Album fields were passed through.
Second attempt: Coherence 0.6.4 on Ubuntu 9.10 in a VM, with the mp3 catalog mounted via NFS. Now the FSStore backend still works, and the MediaDB backend too. This second backend does provide Track/Artist/Album information, and even Album Cover art, downloaded using albumart-qt 1.6.6 on the CentOS server. The Yamaha gets "All Tracks", "Artist", and "Album" listings.
I'll stick with this second solution until I can have the Coherence MediaDB dependencies on CentOS too.

ADSL speed

2010-02-04T13:29:00.003+01:00

ADSL has got me stumped. A couple of months ago, Scarlet (aka Belgacom) claimed that even though my ADSL2 modem synced at around 6Mbps, it wasn't unusual that the maximum downstream data rate was 1Mbps, because of the distance from the modem to the ADSL concentrator apparatus thing. Even though I was very sure that it had been 3+Mbps over a year ago.
Today I notice, from the corner of my eye, the download speed of a virtual appliance I was fetching: 500+ KBps, or a good 5 Mbps. Sync rate of my modem is still 6Mbps.
Was something changed at the Scarlet/Belgacom side ? If yes, why ? Or am I blessed by a unique and unexplained cosmic alignment of some sort ? I just hope it doesn't go down again. Maybe I should keep speedtest.net in my bookmarks.

VMware confirms strategy

2010-01-14T22:37:00.002+01:00

Open source email and calendar solution provider Zimbra got acquired by VMware recently. While there's lots of angles to this story, it also confirms an important part of VMware's strategy: bigger and bigger hardware, running a powerful hypervisor, running thin and optimized OSes, running applications that perform business functions.
To enable perfect mobility and ease-of-use, the OS and application will be packaged into so-called Virtual Appliances or vApps.
Making the OS thinner is essential for this strategy, and goes against the core beliefs of competitor Microsoft. The succes of the vApp concept will be a deciding factor in the battle between these two giants.

10% discount on VCP4 exam

2010-01-09T15:53:00.002+01:00

VMware has announced a 6-month promotion, where students who participated in an official VMware training can get 10% off the regular price of the VCP4 exam, until the end of june 2010. If you plan on doing the VCP4 certification, contact your instructor as soon as possible. He'll give you a personal promotion code that you can use when registering for the VCP4 exam at Pearson Vue. (Yes, that instructor might be me ! ;-))

vmstat limitations

2009-11-12T09:33:00.002+01:00

In the aftermath of the previous post, I ended up monitoring the progress of several huge pvmove and md RAID1 resync operations. Running "vmstat 1" however only shows pvmove block I/O activity, but not md RAID1 resyncing. Fortunately, "dstat -d" shows the reads and writes of both pvmove and md RAID1 resync. ("-d" is automatically included in "dstat -a" by the way.)
N.B. This only affects block I/O measurements, you'll still see the impact on interrupts and CPU statistics in vmstat.

hotplug SATA with CentOS 5

2009-11-07T21:16:00.002+01:00

When the SATA standard was introduced in PCs, I'd read that the electrical connections of both the data and the power connector had been designed with hotplug in mind. But just as with many hotplug-able technologies, I never actually tried it (hotplug PCI, anyone ?). Until now, that is.

Job at hand: replace the software-mirrored SATA drives in high-end Dell workstations with bigger ones. Without losing the data, of course. A perfect opportunity to test how Linux handles SATA hotplug ! Actually, adding the drive was a breeze, Linux automatically detects the drive, and I could sfdisk, and mdadm --add. Next step was hot-removing the old drives: mdadm --fail and mdadm --remove, then physically unplug the drives. I didn't expect it to be so easy, to be honest. Great technology !

you know you're being ripped off when ... (4)

2009-10-24T14:04:00.002+02:00

The familiar packaging of a product gets replaced (or upgraded), the old packaging is no longer available, and the new packaging is priced almost double the original.
Example of the week: 20cl cream at Carrefour. Old brick container, <40c. New brick container with "easy screw-off opening": 69c.
The old packaging must have been easier to recycle and cheaper to make, right ?

audacity: lessons learned

2009-10-17T18:00:00.004+02:00

I had used audacity before, to record a couple of old audio cassettes, cut them into tracks, and burn them on CD (and convert them to .ogg or .mp3). But that was years ago. Last week, I needed audacity again, and had to crawl out of two pitfalls before succeeding.

don't export to a filename that you used or imported while editing. When you open a .wav file, use it in editing, and export, audacity needs the original file to produce the end result. So overwriting the original while audacity still needs it, does not work.
exporting one stereo audio track will convert it to mono for some reason. You'll be left with sound on the left channel, and silence on the right channel. When you split the stereo audio track into two mono audio tracks, you can export the result as a stereo file (.wav, .mp3, whatever you choose).

The rest of the operation was rather simple, and I now have an enhanced digital version of my late grandfather singing, recorded more than 30 years ago !

you know you're being ripped off when ... (3)

2009-10-16T15:56:00.002+02:00

a website pretending to be a search engine, uses your input to fabricate results. Luckily, the ones that I've seen aren't doing much to hide this practice, and searching for "made_up_hokey_pokey" or another randomly invented string will make it very clear.
Example of the day: torrentreactor.to. Every search result under "recently added" is being faked especially for you.

you know you're being ripped off when ... (2)

2009-10-07T11:26:00.002+02:00

the minimal unit used for pricing is actually larger than what you use 99% of the time.
Example of the week: Proximus mobile internet invoiced per 100KB, and my invoice says: 70 connections this month, 7 MB used.

you know you're being ripped off when ...

2009-10-06T21:47:00.003+02:00

...the new packaging of a product is physically larger, but contains less. Check the price, and I bet you it hasn't gone down.
Example of the week: new Duyvis crac a nut packs are 175g, old packs were 200g, but were physically smaller.

The Definitive Guide to CentOS

2009-07-24T12:39:00.004+02:00

Received my paper copy of The Definitive Guide to CentOS today. As the technical reviewer of this book, I've seen the content during the writing phase, but seeing it in print is still cool !

migrating a RAID1 mirror

2009-06-12T19:28:00.002+02:00

Creating a software RAID1 setup with one 2TB and two 1TB drives isn't difficult, but in this case the setup was running in production as a RAID1 over two 1TB drives.
Small sda1 and sdb1 partitions form /dev/md0 and are formatted ext3 for /boot. Large sda2 and sdb2 partitions form /dev/md1 and are an LVM PV, the rest of the system is formatted in logical volumes.
Target situation: to have /boot on all three disks, and the LVM volume group as large as possible, redundant on mirrorred diskspace.
The procedure is strict, but straightforward:

partition the new disk in 3 partitions (small - large - large)
make md0 3-legged with sdc1 as the third device
make md1 3-legged with sdc2 as the third device
wait for the sync (watch /proc/mdstat)
remove sdb2 from md1 and make it 2-legged again (you'll have to --fail sdb2 before you can --remove it).
make a new md2 device for sdb2 and sdc3, make it a PV and add it to your volume group.

The only utilities needed are cfdisk, mdadm, and lvm (for pvcreate and vgextend).
If you need to restructure your existing partition tables in the proces, you might need pvmove to free up existing partitions before reusing or deleting them.

Innovation

2009-06-11T18:01:00.002+02:00

Now that's innovation: automatically detecting the license plate of cars entering a car park / gas station from a highway, and holding them back for a minute if they try to enter the highway too fast. People trying to cut traffic jams (and thereby making them worse, because merging traffic at unadapted speeds creates and worsens traffic jams) get penalized.
Can we get this on the E40 in Heverlee, please ? Pretty please ?

IET to ESX multipathing FAIL

2009-05-29T10:15:00.003+02:00

A software-implemented iSCSI initiator (in VMware ESX) on one side, the iSCSI Enterprise Target (IET on CentOS5) software on the other side, and a separate storage VLAN to connect the two. Works great in my test lab.

But then I wanted to add multipathing to the setup. Plan was: create a second VLAN, and give both the IET server and the iSCSI client a new interface in that VLAN, using a new IP subnet. That gives the client two ways to reach the server, thereby introducing multipathing !

Unfortunately, IET spews error messages at high speed when doing this: I got "kernel: iscsi_trgt: Abort Task (01) issued on tid:1 lun:0 by sid:26459747326427648 (Unknown Task)" about 4000 times per second.

Restarted everything without the second storage VLAN and without the new interfaces, and now all is well again.

Link aggregation between CentOS 5 and a SLM2024

2009-05-15T20:11:00.004+02:00

It's been a while since I made time to try something new. This week, I finally took something off of the "need to try this" list: link aggregation. I've had a gigabit Ethernet switch with link aggregation for about a year now, and my main Linux box has 3 gigE NICs, but I was still only using one. Time for change.

Google found me some good documentation for channel bonding on CentOS5. Manually editing the ifcfg-eth{0,1,2}, ifcfg-bond0, and modprobe.conf is all that's required. That worked, but the default bonding setting is "balance-rr", the simplest loadbalancing algorithm. What I wanted to use was full IEEE 802.3ad link aggregation, mode 4 of the bonding module.

During testing, I got fooled into believing that "service network restart" unloaded and reloaded the bonding module. It doesn't, I should have tested using "service network stop; rmmod bonding; service network start" from the start. Learned my lesson, configured the switch into LACP mode (dynamic link aggregation instead of static), and I was on for some bandwidth testing.

I tried a couple of different bandwidth eaters, but floodping, NFS reads, didn't really stress the configuration. In comes netcat: "nc -l 5555 > /dev/null" on one side, and "nc myserver 5555 < /dev/zero" on the other, and you'll get a gigabit stream of data in no-time. Using dstat and a couple of netcats, the current record stands at more than 200 MBps. Mission accomplished !

Watching dd progress

2009-02-26T18:47:00.003+01:00

Fabian (of RPMforge and CentOS fame) wrote a blogpost about using pv (pipe viewer) to watch a datastream through a pipe, such as a dd-over-ssh copy from one machine to another. I knew pv existed, but I've often used dd on systems where pv wasn't available (e.g. rescue CD), or used dd without sending data through pipes. Even in these situations, you can keep an eye on dd's progress: open another terminal window, then run

# while sleep 10; do killall -USR1 dd; done

and switch back to the shell where dd is running. It will show its status every 10 seconds.
Note, your version of dd needs to support this. The dd from RHEL3 doesn't (version 4), but the dd from RHEL5 does (version 5) !

scoop: VMware vCenter on CentOS 5

2009-02-25T15:39:00.005+01:00

Today at VMworld Europe 2009, VMware demoed a preview of their vCenter management server running on Linux in a virtual machine, pre-packaged in virtual appliance format. Aside from that fact that this is a great idea with all the advantages that virtual appliances have to offer, and that a lot of people have been waiting for, it was a really pleasant surprise to me that they chose to build it on top of CentOS 5 !
It'll take months before a final version will become available for download to VMware vCenter customers, but it's going to happen, that's for sure.
Kudos to the CentOS team !

vCenter forgot my ESX is in standby mode. What now ?

2009-02-07T12:01:00.002+01:00

I used VirtualCenter (vCenter) to put an ESX in standby mode. I restarted VirtualCenter for Windows patches, and now VirtualCenter says the ESX host is "not responding", and doesn't give me the option to wake it from standby mode. Fortunately, the solution is right there under your fingertips: do manually what VirtualCenter would have done for you:
Log on to another ESX on the same network, get the MAC address(es) of the sleeping ESX host, and run:
# ether-wake -i vswif0 aa:bb:cc:dd:ee:ff
You could optionally repeat this for every interface you know of the sleeping ESX. Remember to use the real MAC addresses of the real NICs, not the virtual MAC addresses of service console ports, vmkernel ports, or virtual machines.
After a couple of minutes, VirtualCenter will notice that the host is responding, and will connect it again. If you wish, you can put it in standby mode again using the VI-Client.

If you don't know the MAC address of the host in standby mode, you would need to look for it in the VC database, but you can also find it in /etc/vmware/esx.conf, if you happen to have a backup (such as a vm-support dump). Look for /net/pnic/child[*]/mac values.

autovacuum in the real world

2009-01-09T12:38:00.006+01:00

Any sysadmin who runs postgresql (or sqlite) knows he should vacuum once in a while. You didn't ? Probably because nowadays, there's an autovacuum function. But what about the Real World (you know, the big room with the blue ceiling) ?

After years of manually vacuuming the floors, I now got myself an autovacuumer to take care of my office space: the iRobot Roomba 560. This robot can drive around on the floor and cleans it. After having cleaned the room, it automatically returns to the dock where it recharges the battery.

Aside from the "cool" factor, it really does a good job. I've seen comments from people who claim that a vacuum cleaner can't run on batteries for an hour, even proving their point with calculations, but don't be fooled: the Roomba is not a vacuum cleaner like the one you already own. First, doesn't suck dirt through a 3 meter tube. Second, it isn't even a vacuum cleaner in the strictest sense: it uses brushes to pick up most of the dirt, then using a small DustBuster style vacuum engine for the finishing touch.

Overall verdict: I should have bought one sooner !

getting rid of numeric codes in mc2xml output

2008-12-23T20:26:00.004+01:00

mc2xml is a TV schedule grabber that can be used for MythTV. The output of mc2xml (why isn't this program opensource, by the way ?) is in the xmltv format, but the channel names are rather cryptic. I don't want to see "I147350.750834.microsoft.com", but rather "ned 3". Getting rid of these codes is easy with a small shell script.
mc2xml -c be -g 3000
cat ~/channelconvert.txt | while read line
do
sed -e "s/$line/g" -i xmltv.xml
done
Where channelconvert.txt contains lines such as
I343250.54932.microsoft.com/cnni
I399250.70.microsoft.com/bbc1
Then input it in your database with mythfilldatabase. Happy pvr-ing !

goosync works

2008-12-16T23:07:00.002+01:00

Last week, I decided to buy a goosync subscription. I thought about this for a long time, and now a Christmas special convinced me. And honestly, I shouldn't have postponed it. Setup on my E71 was so easy I was amazed it worked as quickly as it did. Talking about my availability with my clients without firing up my laptop and connecting it to the network to open my Google Calendar, that's what I bought a smartphone for.
There are a couple of improvements that could be made, like not requiring labels for every single calendar that needs to be synced (why add a label to a week-numbering calendar, or a public holiday calendar ?).
Allround verdict: great service !

The myth of the sudoku's

2008-10-27T15:26:00.002+01:00

Sudoku's have been quite a hype in recent years, attracting puzzle lovers all over the world. I have solved quite a number of them myself, or at least, that's what I thought until earlier this year. Seeing the daily "difficult"/"superieur"-level sudoku in a Swiss free newspaper, a coin dropped. They've been publishing the same sudoku for over a year now. Yes, that's right, the same sudoku, every day. Or more precisely, an equivalent sudoku. By using four simple transformations, yesterday's sudoku can be transformed into today's, and so can the solution !

rearranging rows withing the same 3-row block (idem with columns)
rearranging the order of the 3-row blocks (idem with 3-column blocks)
permutating 123456789 into a different order
transposing the grid (making rows into columns and vice versa)

How long can they keep this up, you wonder ? There's a total of about 1.2x10E12 combinations to every sudoku puzzle. It will take them 4.6 billion years before they run out of variations of this "one and only" sudoku.

PS the title of this article refers to "the myth of the fingerprints" by Paul Simon. "I've seen them all and man, they're all the same..."

identifying the ESX version in a VM: peeking through the blinds

2008-10-01T21:29:00.012+02:00

An interesting issue popped up: how can a program or script in a VM determine the type and version of the hypervisor it is running on ? This is important if you want to install version-specific software, like the VMware Tools ("enlightened hardware drivers", as MS would say).
Knowing that you're running in a virtual machine is pretty easy: you can look at the hardware, on VMware you'll see "00:0f.0 VGA compatible controller: VMware Inc Abstract SVGA II Adapter" which makes it pretty clear, and lspci -v says "Subsystem: VMware Inc" for every PCI device. Easy as 1 2 3.

But which version of VMware ESX is it ? Not that easy, it seems. But we (hi Dag !) found a unique signature for each ESX version: looking at dmidecode, the output in the "BIOS Information" section lists a Release Date and an Address field. These seem to be unique for each ESX version. The mapping we could build was:
ESX 2.5 - BIOS Release Date: 04/21/2004 - Address 0xE8480 - Size 97152 bytes
ESX 3.0 - BIOS Release Date: 04/17/2006 - Address 0xE7C70 - Size 99216 bytes
ESX 3.5 - BIOS Release Date: 01/30/2008 - Address 0xE7910 - Size 100080 bytes
ESX 4(?) - BIOS Release Date: 08/15/2008 - Address 0xEA6C0 - Size 88384 bytes

Tested versions: ESX 2.5.2, 3.0.2, and 3.5u2. We received an anonymous tip about ESX 4 beta results. Thanks !
VMware Workstation 4.5 reports the same as ESX 2.5. VMware Server 1.0 reports the same as ESX 3.0. Workstation 6.5 reports the same as ESX 4(?). Workstation 6.0 and Server 2.0.0 report
WS 6.0 - BIOS Release Date: 12/06/2006 - Address 0xE78A0 - Size 100192 bytes
SRV 2.0.0 - BIOS Release Date: 7/29/2008 - Address 0xEA6C0
but we didn't find an ESX version that reported the same values. Refer to the VMware "mobility guide" document for other possible relationships between different VMware products.

P.S. 26/11 thanks to Michael for sending us information on VMware Server 2.0.0 ! Same address as WS 6.5 and ESX4(?), but different release date.

sports tracker on the nokia e71

2008-09-21T23:05:00.004+02:00

Lots of people seem to experience freezes of Nokia Sports Tracker on their E71 smartphone. Google easily finds articles suggesting a workaround: "don't leave Nokia Sports Tracker in the foreground". However, that doesn't help. What does help, is setting the access point to "None" in the Nokia Sports Tracker settings. The program doesn't like an inaccessible access point in its preferences.

a plea for non-identical mirrors

2008-09-05T16:58:00.004+02:00

When I set up my test environment earlier this year, I put two Samsung HD103UJ 1TB drives in the main file server, and configured them as a RAID-1 software mirror through mdadm. Just months later, smartd was spewing out messages about both drives: Device: /dev/sda, 8 Currently unreadable (pending) sectors.

That got me very worried... I replaced one of the drives with a WDC WD1000FYPS-01ZKB0, and rebuilt the mirror. Actually, I built new mdadm mirrors from smaller partitions to lessen the risk of a two-drive badblock scenario. Regardless, my data got moved, so now everything gets stored on two different drives. Different makes, manufacturers, models. Same size though.

The problem is, I can't RMA the drive. It's still in warranty, but when running badblocks -w on the drive, it shows no problems whatsoever. Moreover, the SMART status has gone back to: Current_Pending_Sector 0. So not only is the problem undetectable by badblocks, it even makes it go away in the SMART counters where it was visible in the first place.
First I thought that the problematic sectors got reallocated, but the SMART info suggests otherwise: Reallocated_Event_Count equals 0. If you want to look at your own drives, grep for _Pending_ and Reallocated_ in the smartctl -a output.

My conclusion: it was a very, very bad idea to make a mirror out of two identical drives (and aggravating the fact, they were the cheapest ones, too). Please remember: pick different drives to make up a mirror. A mirror image needs to be identical, but having it on identical media is a bad, bad thing.

And to round up, my final thoughts: how bad is a non-zero "Current_Pending_Sector" count, really ? And why can it go down without reallocation ?

Side note: I'm not saying Samsung drives are bad, I have 3 Samsung HD300LJ in a RAID-1 setup that are doing fine after more than 2 years of 24x7 service. But buying their 1TB drives left a sour aftertaste...

don't forget mkinitrd

2008-08-08T14:59:00.002+02:00

Most Linux system-administrators are well aware of the benefits of LVM. With online resizing of filesystems and migration of data from one disk to another, it's fantastic. But don't assume that your system will do everything for you.
If your system has one disk, that you used as a physical volume (PV) in a volume group (VG), where your root partition is stored as a logical volume (LV), you can easily add a new disk. Add the disk physically, boot up, use pvcreate and vgextend to include the new disk in the existing volume group. Just don't reboot, at least, not yet !

Your system requires a manual rebuild of its initial ram disk
/sbin/mkinitrd -f /boot/initrd-2.6.18-92.1.10.el5.img 2.6.18-92.1.10.el5

If you forget to do this, your system will not boot, because it won't find all the components necessary to activate the VG it needs to access the root filesystem. Symptoms: kernel loads, initrd loads, root filesystem can't be mounted because the volume group doesn't exist.
If it happened already (and that's why google sent you to this blog entry), don't despair, it's an easy thing to fix with the "linux rescue" option of your RHEL/CentOS installation CD or DVD.

how many physical hosts do you buy: what MS sales didn't tell you

2008-08-08T10:20:00.007+02:00

One of the first steps in a virtualization project is building a list of workloads that will get virtualized, with a measurement or estimate of the resources that they will need. X MHz and Y MB, sum everything up, and let's say you get 30 GHz of CPU power and 20 GB of RAM.

The hardware you'd like to run all those virtual machines on can handle two CPUs (dual-socket), four cores each (quad-core). That means that every physical server will give you between 20 and 25 GHz of CPU power. For memory, you'll buy 12 GB of RAM in each server.

So the plan is to buy two of those servers, right ?

Well, as long as your infrastructure is 100% healthy and running OK, two servers will do the job just fine. You've got enough resources, with a bit of headroom for overhead and future growth. But what happens when one of the physical servers is down ? Think of hardware problems, think of virtualization software upgrades, think of patching the hypervisor.

Then the available resources are down to 20 GHz and 12 GB of RAM. For CPU, 20 GHz means that every application will get 30% less than desired, and will therefore run a bit slow, probably noticable for users. Is that acceptable in these cases ?

And last but not least, memory. Temporarily, you've got just 12 GB of RAM available, and your VMs need 20 GB. Did you know that what happens depends on which hypervisor you've chosen ?

With Hyper-V or Xen, you're in trouble. With 12 GB, you can run 60% of your VMs, and the rest stays down.
With ESX and ESXi, you can start all your VMs, and just as with CPU, there's not enough resources so everything will slow down a bit. But it will run. This trick is called "memory overcommitment".

Conclusion: if you chose Hyper-V or Xen, you'll need three of those servers to continue running your business. With VMware, you have the option of buying three and continuing without speed impact, or you just buy two servers, and live with the temporary loss of performance.

Now let me guess, did your Microsoft sales guy tell you about this difference ?

are you still using RHEL 2.1 ?

2008-08-04T16:34:00.003+02:00

Are you still using Red Hat Enterprise Linux 2.1, or CentOS 2.1 ? Then this news is of great importance to you: planned End-Of-Life for these products is approaching ! After May 2009, there will be no more security updates for RHEL 2.1, nor support from Red Hat.

It's not really news, as the lifecycle of RHEL products has always been clearly announced and published. If you missed all that, this is the time to start planning an upgrade. Your Red Hat subscription gives you the right to use the newer versions of RHEL, so upgrading is all you need to do.

Need help planning an upgrade to RHEL 3, 4 or 5 ? I can recommend some experienced consultants ! ;-)

choosing hardware for an ESX testlab

2008-08-01T14:38:00.003+02:00

When you're shopping for ESX servers to build your next production cluster, you know where to look. The hardware compatibility lists at vmware.com are frequently updated, and contain everything you need to know. But what about test labs ? When you don't care too much about "is it supported" but rather ask yourself "does it work" and "can I buy something cheaper" ?

I asked myself the same questions when I built my own testlab. And this was my choice: 2 identical PCs, equipped with

MSI MS7345 motherboard
Q6600 intel CPU
8 GB of RAM (four times 2 GB)
PCI Promise SATA 300 TX2plus
80 GB sata harddrive (smallest I could get)
a dual port Intel gigabit card

Add a third PC, similar but with more diskspace, running CentOS 5 with IET as an iSCSI server, and you get a 19 GHz, 16 GB VI3 cluster with several hundred GB storage space. Perfect for testing, and without losing an arm and a leg paying for them.

how do I enable EVC when VC is running in a VM ?

2008-08-01T14:14:00.002+02:00

So you're running the new ESX 3.5 update 2. And you want to try the new EVC feature on your cluster. And you've found it under "edit settings" on your cluster object...
And then VirtualCenter refuses to enable it for you, because there is still at least one VM powered on in the cluster. Of course there is, because your VirtualCenter runs in a VM, in that same cluster. Catch-22 ?

Well, getting round this is a bit weird, but nevertheless possible. Here's what I did on my two-node test cluster:
Step one: you evacuate one ESX host in your cluster. You put it in maintenance mode, and move it out of the cluster, right under your datacenter object.
Two: manually migrate (VMotion via drag-and-drop for example) your VirtualCenter VM (running on a host in the cluster) to the host that is now outside the cluster. Do this with all other VMs that were still running in the cluster.
Three: enable EVC on your cluster. This now works, because the cluster doesn't contain any running VMs anymore.
Four: migrate the VMs back to a host within the cluster (again, drag and drop)
And finally: put the host back into the cluster.