Saturday, July 19, 2014

GEM WS2 MIDI System Exclusive structure and checksums

MIDI is the standard for communication between electronic music instruments like keyboards and synthesizers. And computers! While tinkering with an old floppy-less GEM WS2 keyboard, I wanted to figure out the structure of their System Exclusive memory dumps. SysEx is the vendor-specific (and non-standard) part of MIDI. Vendors can use it for real-time instructions (changing a sound parameter in real-time) and for non-real-time instructions (sending or loading a configuration, sample set, etc.).

In the GEM WS2, there's two ways of saving the memory (voices, globals, styles and songs): in .ALL files on floppy, and via MIDI SysEx.

The .ALL files are binary files, 60415 bytes long. The only recognizable parts are the ASCII encoded voice and global names. The SysEx dumps are 73691 bytes long. As always in MIDI, only command start (and end) bytes have MSB 1, and all data bytes have MSB 0. The data is spread out over 576 SysEx packets, preceded by one SysEx packet with header information.

Each SysEx data packet starts with these bytes (decimal representation):

  • 240 (SysEx start) 
  • 47 (GeneralMusic / GEM / Elka manufacturer ID) 
  • 2 (the header packet has a 1 here, the data packes have a 2) 
  • a six-bit packet counter (data packet number MOD 64)
  • 15 (data length, discussed below)
  • then there's room for 120 data bytes
  • one checksum byte (discussed below)
  • 247 (SysEx end)


Because the original data (the WS2 memory and the .ALL file) has 8 bits per byte, and MIDI SysEx bytes can only have 7 bits (MSB 0), GEM uses an encoding to go from one to the other:
Seven 8-bit bytes have their LSB stripped, and the LSB's form byte number 8, from the first of seven bytes in the LSB of byte number 8, to the last of seven bytes in bit number 7 (128 decimal value).
Using this encoding, a group of 7 bytes from the .ALL format is transformed into a group of 8 SysEx bytes.

The length byte in each data packet indicates how many of those byte groups there are in the current data packet. Data is sent per 15 byte groups., resulting in a 127 byte SysEx packet, with the last data packet containing the remaining 6 byte groups. There's only five bytes in the .ALL format to fill the last byte group of the last data packet, and that byte group is padded with two FF(255) bytes.

The checksum byte is calculated as the XOR of all other bytes in the SysEx data packet, excluding the 240 and 247 start and stop bytes. When receiving a SysEx dump, the total XOR checksum of the bytes between 240 and 247 should therefore always be 0. (NB this is substantially different from the Roland way of doing SysEx checksums).

With this knowledge, I wrote a Perl script to convert .ALL files to SysEx (known as .syx) bytestreams. Owners of GEM WS1/WS2/WS400 keyboards who find themselves without floppies or without a working floppy drive can now load their .ALL files via a computer (with e.g. MIDI-OX or SysEx Librarian). If interested, send me an e-mail!




Saturday, December 14, 2013

Identifying virtual disks in Linux on vSphere

A default virtual machine has straightforward hardware. A single SCSI disk on a single SCSI card, for example. Having multiple SCSI disks or cards in a VM creates the need for in-guest identification. Linux complicates matters slightly by using alphabetical disk naming: /dev/sda, /dev/sdb, ... /dev/sdz, /dev/sdaa, /dev/sdab, ... This post looks at how you can identify individual disks in a VMware virtual machine.

Executive summary: VMware notation "X:Y" typically maps onto Linux scsi(X+2), Id:Y, which are then named in ascending order with /dev/sd* identifiers.

which vSphere version is my VM running on?

(an update of an older post, now complete up to vSphere 5.5)

Your Linux runs on a VMware VM, but which on which ESXi version? You can see for yourself: run "dmidecode" and look at lines 10, 11 and 12.
ESX 2.5 - BIOS Release Date: 04/21/2004 - Address 0xE8480 - Size 97152 bytes
ESX 3.0 - BIOS Release Date: 04/17/2006 - Address 0xE7C70 - Size 99216 bytes
ESX 3.5 - BIOS Release Date: 01/30/2008 - Address 0xE7910 - Size 100080 bytes
ESX 4 - BIOS Release Date: 08/15/2008 - Address 0xEA6C0 - Size 88384 bytes
ESX 4U1 - BIOS Release Date: 09/22/2009 - Address 0xEA550 - Size 88752 bytes
ESX 4.1 - BIOS Release Date: 10/13/2009 - Address 0xEA2E0 - Size 89376 bytes
ESXi 5 - BIOS Release Date: 01/07/2011 - Address 0xE72C0 - Size 101696 bytes

ESXi 5.1 - BIOS Release Date: 06/22/2012 - Address: 0xEA0C0 - Size: 89920 bytes
ESXi 5.5 - BIOS Release Date: 07/30/2013 - Address: 0xEA050 - Size: 90032 bytes

NB These DMI properties are set at boot time. Even if your VM gets live-migrated to a host running a different vSphere version, your VM will keep the values it got from the host it booted on.

Saturday, November 9, 2013

VMware Certified Associate exams half-price until 31/1/2014

Using a voucher code, anyone can do a VCA exam (datacenter virtualization, cloud, or desktop, and maybe network when it becomes available) for half the regular price. VCA is a certification that you can do from the comfort of your desk! Enroll at http://vmware.com/certification/ . To do the exam for half the normal price, use this voucher code: VMRT4B425324.
Best of luck to anyone who tries!

NB This code used to reduce the exam price to 0, rendering VCA certification completely free, but starting somewhere in december 2013, the discount will be reduced to 50%.

Monday, September 30, 2013

downloading protected/embedded videos - the hard way

Charged with the task to save a news broadcast for posteriority, I wanted to download a video that was only available through an embedded "JW Player" videoplayer on a website. Time was not on my side: the video would be purged from the site within a week.

In the HTML source of the webpage containing the video, I found tags like
data-video-src="http://media.XYZnieuws.net/GEO2013/09/230916509ONL1309244843676.GeoFLVLong.flv"
data-video-iphone-path="GEO2013/09/230916509ONL1309244843676.GeoMP4_H.264.m4v
or (from another video)
data-video-rtmp-path="2009/11/132628352ONL0911177754876.urlFLVLong.flv"
but I couldn't get a clear URL out of the page's source code (URLs anonymized to protect involved parties).
After a bit of research (a.o. channeling browser traffic through a proxy and looking at the URLs being requested), the files used by the JW Player were still a mystery to me, but I found the server that was hosting the iphone video files: iphone.WXYpower.be. And that's where I got lucky: requesting http://iphone.WXYpower.be/XYZnieuws_geomo/_definst_/GEO2013/09/230916509ONL1309244843676.GeoMP4_H.264.m4v/chunklist.m3u8
resulted in a list of media_N.ts files (where N is a number starting from 1, and in this case ranging to 169).
Fetching all those files (wget) and concatenating them (cat) into one big file, gave me a large mpeg file, playable in VLC. Converting it to mp4 with handbrake reduced it to 75% its original size.

Thursday, July 25, 2013

SSD overprovisioning in vSphere

SSD vendors like Samsung or Intel often provide tools to reserve some space on your SSD that can be used by the internal algorithms for better wear-leveling, longevity, garbage collection, and performance. My own Samsung SSD comes with a tool to do that on Windows, for example.
But on vSphere, you'll have to do that manually: I see two possibilities:

  1. don't format the SSD "full" but instead use "partial", and specify only the amount of the disk that you want to use. The rest is left unpartitioned, and therefore unallocated. You can only do this if you haven't formatted the SSD as VMFS yet. If you have, this second possibility can still help you:
  2. create a thick-provisioned lazy-zeroed VMDK on the SSD-backed VMFS, either with vmkfstools or by creating a dummy VM. Those blocks are allocated, but remain unwritten. You're not going to attach the disk to a running VM, so they won't ever be written to. This effectively reduces the amount of blocks that can be written to, leaving the rest open. Caveat: if you've previously removed VMDKs from the volume, you have no guarantee that the underlying blocks are in an unwritten state.

P.S. Your SSD might have some overprovisioning space built in already. I've always assumed that's the reason some vendors sell 256GB while others sell 240GB models. The latter vendors sell the same amount of flash as the former, but leave 1/16th unallocated.

Tuesday, July 23, 2013

dd-wrt refusing new configuration: out of NVRAM space

Another post documenting a small issue just so I can google it later: one of the Cisco/Linksys WRT160NL devices (both running DD-WRT) in my environment stopped accepting config changes through its web interface. It happened to be the one I'm using as a NAT gateway and DHCP server, not the one just functioning as access point.
A config backup appeared to be full of UPnP rules. Those are port forwarding rules that have been asked for by applications on NATed stations that want to be reachable. Think Skype amongst many others.
Clearing the UPnP list solved the problem: click "NAT/QoS", then "UPnP", then "Delete all". Apparently the UPnP list filled up the available NVRAM space, which broke all subsequent config changes.