Saturday, March 27, 2010

Cisco SLM2024 and Firefox

I don't know what got me to search for new firmware for my Linksys (now Cisco) SLM2024. But I did, and found version (upgrade from the previous 1.0.1). Aside from the web interface, there's no obvious changes. It is now branded Cisco, rather than Linksys. But the most important change for me wasn't in the release notes: I can now manage my gigabit switch using Firefox ! (tested with Firefox 3.6.2 on Windows)

Friday, March 5, 2010

HZ divider effect on timer interrupt overhead

Red Hat and related distro's (like CentOS) use 1000 timer interrupts per second, per CPU core or thread (this is called the "HZ value" inside the kernel). Because this causes a lot of extra work in case of virtualization, and caused (past tense since RHEL 5.4!) problems with timekeeping, the "divider" kernel parameter has been introduced. For example, by booting with "divider=10", the kernel uses 100 timer interrupts instead of 1000, and "divider=25" means 40 timer interrupts per second.
I did a little test today to see what difference that makes when running CentOS5.4 on vSphere. Tests were done with the current 2.6.18-164.11.1.el5 x86_64 kernel in a single vCPU VM. These are the results from my test environment:
  • standard settings: HZ=1000: 60 MHz cycles used when idle.
  • divider=2: HZ=500: 36 MHz cycles used when idle.
  • divider=5: HZ=200: 25 MHz cycles used when idle.
  • divider=10: HZ=100: 20 MHz cycles used when idle.
  • divider=25: HZ=40: around 10-15 MHz cycles used when idle (results slightly variable, also for higher dividers)
  • a nohz "tickless" kernel (I used, not currently available in RHEL/CentOS by default) used 70 MHz cycles when idle in my test environment. This is a surprise, and I don't have a good explanation (yet) !
Note that 40 is the maximum divider setting. Using 41 will show "tick_divider: 41 is out of range" in dmesg, and the parameter will be ignored.
On your own machines, you can easily check:
  • the current divider setting by looking at the currently active kernel parameters: cat /proc/cmdline . No divider parameter means divider=1, the default.
  • the measured timer interrupt rate (close approximation of HZ value): dstat -i -I timer
Gaining 20-40 MHz of CPU power on your virtualization host might not seem important, but this is a per-guest win ! Do this on all your VMs, and you could be freeing several hundreds of MHz per host, and get more VMs, better consolidation ratio's, better performance, or lower power usage.

Summary: before RHEL 5.4, divider=10 was recommended for timer accuracy. This is no longer true, but as I've shown, it still helps lowering the timer interrupt overhead. Don't forget that the ideal divider setting depends on your application: thread wake-up delays can occur in high divider scenario's, and responsiveness could potentially suffer because of that.