Wednesday, June 22, 2016

How VMware appliances update themselves

Most VMware appliances (vCenter Appliance, VMware Support Appliance, vRealize Orchestrator) have the so called VAMI: the VMware Appliance Management Interface, generally served via https on port 5480. VAMI offers a variety of functions, including "check updates" and "install updates". Some appliances offer to check/install updates from a connected CD iso, but the default is always to check online. How does that work?

VMware uses a dedicated website to serve the updates: Each appliance is configured with a repository URL: . The PRODUCT-ID is a hexadecimal code specific for the product. vRealize Orchestrator uses 00642c69-abe2-4b0c-a9e3-77a6e54bffd9, VMware Support Appliance uses 92f44311-2508-49c0-b41d-e5383282b153, vCenter Server Appliance uses 647ee3fc-e6c6-4b06-9dc2-f295d12d135c. The VERSION-ID contains the current appliance version and appends ".latest":,,

The appliance will check for updates by retrieving the repository URL /manifest/manifest-latest.xml . This xml contains the latest available version in and (fullVersion includes the build number), pre- and post-install scripts, EULA, and a list of updated rpm packages. Each entry has a that can be appended to the repository URL and downloaded. The update procedure downloads manifest and rpms, verifies checksums on downloaded rpms, executes the preInstallScript, runs rpm -U on the downloaded rpm packages, executes the postInstallScript, displays the exit code and prompts for reboot.

With this information, you can setup your own local repository (for cases where internet access is impossible from the virtual appliances), or you can even execute the procedure manually. Be aware that manual update would be unsupported. Using a different repository is supported by a subset of VMware appliances (e.g. VCSA, VRO) but not all (VMware Support Appliance).

Thursday, December 3, 2015

VPN gateway setup for Android 5, iOS 9, and Mac OS X 10.10

I recently configured an IKEv1 L2TP/IPSec VPN for a customer. They needed support for a mix of Android 5, iOS 9, and Mac OS X 10.10 clients. During testing and going through debug logs on the VPN gateway, I found that these devices announce support for several authentication hashes, and encryption protocols:
Android 5SHA256-128, SHA1-96, MD5-96AES256, AES128, 3DES, DES
iOS 9SHA1-96, MD5-96AES256, AES128, 3DES
Mac OS X 10.10SHA1-96, MD5-96AES256, AES128, 3DES

The working configurations I found were:

and I settled on the last combo as AES256 is the strongest CBC from that list.

PS for DH key exchange, only so-called Group 2 1024modp was in the list on all three devices, so there was no other choice available, and no further testing was done.
PS2 I tried SHA256 authentication with the Android device, but no successful connection could be set up with the VPN gateway. It looks like there was some kind of incompatibility between the SHA256 implementations on both devices. As the Apple devices didn't announce support for SHA256, there was no reason to debug that in this environment.
PS3 Some of the acronyms encountered during these tests: IKE, HMAC, PRF, CBC

Tuesday, September 1, 2015

A use case for exporting and importing distributed vswitches

In a recent VMware project, an existing environment of vSphere ESXi hosts had to be split off to a new instance of vCenter. These hosts were member of a distributed virtual switch, an object that saves its configuration in the vCenter database. This information would be lost after the move to the new vCenter, and the hosts would be left with "orphaned" distributed vswitch configurations.

Thanks to the export/import function now available in vSphere 5.5 and 6.x, we can now move the full distributed vswitch configuration to the new vCenter:

  • In the old vCenter, right-click the switch object, click "Export configuration" and choose the default "Distributed switch and all port groups"
  • Add the hosts to the new vCenter
  • In the new vCenter, right-click the datacenter object, click "Import distributed switch" in the "Distributed switch" sub-menu.
  • Select your saved configuration file, and tick the "Preserve original distributed switch and port group identifiers" box (which is not default!)
What used to be orphaned configurations on the host, are now valid member switches of the distributed switch you just imported!

Monday, June 29, 2015

How to solve "user locked out due to failed logins" in vSphere vMA

In vSphere 6, if the vi-admin account get locked because of too many failed logins, and you don't have the root password of the appliance, you can reset the account(s) using these steps:

  1. reboot the vMA
  2. from GRUB, "e"dit the entry
  3. "a"ppend init=/bin/bash
  4. "b"oot
  5. # pam_tally2 --user=vi-admin --reset
  6. # passwd vi-admin # Optional. Only if you want to change the password for vi-admin.
  7. # exit
  8. reset the vMA
  9. log in with vi-admin
These steps can be repeated for root or any other account that gets locked out.

If you do have root or vi-admin access, "sudo pam_tally2 --user=mylockeduser --reset" would do it, no reboot required.

Tuesday, June 2, 2015

VCSA detailed sizing options

The vCenter Server Appliance in vSphere6 can be deployed as "tiny", "small", "medium", and "large". The deployment wizard gives info about the vCPU and vRAM consequences of this choice, and about the total disk size of the appliance. But as there's 11 (eleven!) disks attached to the VCSA appliance, it's worth looking into which disks get a different size.

Max hosts10100400
Max VMs10010004000
Disk size120150300
disk0/ and /boot121212

N.B. I currently don't have the config data for "large".
This table can help if your environment is growing slightly or wildly beyond the original sizing of the VCSA. Using the autogrow command in @lamw's article, you can easily grow the relevant parts of the VCSA VM.

Tuesday, May 26, 2015

which vSphere version is my VM running on?

(an update of an older post, now complete up to vSphere 6)

Your Linux runs on a VMware VM, but which on which ESXi version? You can see for yourself: run "dmidecode" and look at lines 10, 11 and 12.
ESX 2.5 - BIOS Release Date: 04/21/2004 - Address 0xE8480 - Size 97152 bytes
ESX 3.0 - BIOS Release Date: 04/17/2006 - Address 0xE7C70 - Size 99216 bytes
ESX 3.5 - BIOS Release Date: 01/30/2008 - Address 0xE7910 - Size 100080 bytes
ESX 4 - BIOS Release Date: 08/15/2008 - Address 0xEA6C0 - Size 88384 bytes
ESX 4U1 - BIOS Release Date: 09/22/2009 - Address 0xEA550 - Size 88752 bytes
ESX 4.1 - BIOS Release Date: 10/13/2009 - Address 0xEA2E0 - Size 89376 bytes
ESXi 5 - BIOS Release Date: 01/07/2011 - Address 0xE72C0 - Size 101696 bytes

ESXi 5.1 - BIOS Release Date: 06/22/2012 - Address: 0xEA0C0 - Size: 89920 bytes
ESXi 5.5 - BIOS Release Date: 07/30/2013 - Address: 0xEA050 - Size: 90032 bytes
ESXi 6 - BIOS Release Date: 09/30/2014 - Address: 0xE9A40 - Size: 91584 bytes

NB These DMI properties are set at boot time. Even if your VM gets live-migrated to a host running a different vSphere version, your VM will keep the values it got from the host it booted on. What you see is the vSphere version of the host your VM booted on.

Saturday, July 19, 2014

GEM WS2 MIDI System Exclusive structure and checksums

MIDI is the standard for communication between electronic music instruments like keyboards and synthesizers. And computers! While tinkering with an old floppy-less GEM WS2 keyboard, I wanted to figure out the structure of their System Exclusive memory dumps. SysEx is the vendor-specific (and non-standard) part of MIDI. Vendors can use it for real-time instructions (changing a sound parameter in real-time) and for non-real-time instructions (sending or loading a configuration, sample set, etc.).

In the GEM WS2, there's two ways of saving the memory (voices, globals, styles and songs): in .ALL files on floppy, and via MIDI SysEx.

The .ALL files are binary files, 60415 bytes long. The only recognizable parts are the ASCII encoded voice and global names. The SysEx dumps are 73691 bytes long. As always in MIDI, only command start (and end) bytes have MSB 1, and all data bytes have MSB 0. The data is spread out over 576 SysEx packets, preceded by one SysEx packet with header information.

Each SysEx data packet starts with these bytes (decimal representation):

  • 240 (SysEx start) 
  • 47 (GeneralMusic / GEM / Elka manufacturer ID) 
  • 2 (the header packet has a 1 here, the data packes have a 2) 
  • a six-bit packet counter (data packet number MOD 64)
  • 15 (data length, discussed below)
  • then there's room for 120 data bytes
  • one checksum byte (discussed below)
  • 247 (SysEx end)

Because the original data (the WS2 memory and the .ALL file) has 8 bits per byte, and MIDI SysEx bytes can only have 7 bits (MSB 0), GEM uses an encoding to go from one to the other:
Seven 8-bit bytes have their LSB stripped, and the LSB's form byte number 8, from the first of seven bytes in the LSB of byte number 8, to the last of seven bytes in bit number 7 (64 decimal value).
Using this encoding, a group of 7 bytes from the .ALL format is transformed into a group of 8 SysEx bytes.

The length byte in each data packet indicates how many of those byte groups there are in the current data packet. Data is sent per 15 byte groups., resulting in a 127 byte SysEx packet, with the last data packet containing the remaining 6 byte groups. There's only five bytes in the .ALL format to fill the last byte group of the last data packet, and that byte group is padded with two FF(255) bytes.

The checksum byte is calculated as the XOR of all other bytes in the SysEx data packet, excluding the 240 and 247 start and stop bytes. When receiving a SysEx dump, the total XOR checksum of the bytes between 240 and 247 should therefore always be 0. (NB this is substantially different from the Roland way of doing SysEx checksums).

With this knowledge, I wrote a Perl script to convert .ALL files to SysEx (known as .syx) bytestreams. Owners of GEM WS1/WS2/WS400 keyboards who find themselves without floppies or without a working floppy drive can now load their .ALL files via a computer (with e.g. MIDI-OX or SysEx Librarian). If interested, send me an e-mail!