GEM WS2 MIDI System Exclusive structure and checksums

MIDI is the standard for communication between electronic music instruments like keyboards and synthesizers. And computers! While tinkering with an old floppy-less GEM WS2 keyboard, I wanted to figure out the structure of their System Exclusive memory dumps. SysEx is the vendor-specific (and non-standard) part of MIDI. Vendors can use it for real-time instructions (changing a sound parameter in real-time) and for non-real-time instructions (sending or loading a configuration, sample set, etc.).

In the GEM WS2, there's two ways of saving the memory (voices, globals, styles and songs): in .ALL files on floppy, and via MIDI SysEx.

The .ALL files are binary files, 60415 bytes long. The only recognizable parts are the ASCII encoded voice and global names. The SysEx dumps are 73691 bytes long. As always in MIDI, only command start (and end) bytes have MSB 1, and all data bytes have MSB 0. The data is spread out over 576 SysEx packets, preceded by one SysEx packet with header information.

Each SysEx data packet starts with these bytes (decimal representation):

• 240 (SysEx start) 
• 47 (GeneralMusic / GEM / Elka manufacturer ID) 
• 0 
• 2 (the header packet has a 1 here, the data packes have a 2) 
• a six-bit packet counter (data packet number MOD 64)
• 15 (data length, discussed below)
• then there's room for 120 data bytes
• one checksum byte (discussed below)
• 247 (SysEx end)

Because the original data (the WS2 memory and the .ALL file) has 8 bits per byte, and MIDI SysEx bytes can only have 7 bits (MSB 0), GEM uses an encoding to go from one to the other:
Seven 8-bit bytes have their LSB stripped, and the LSB's form byte number 8, from the first of seven bytes in the LSB of byte number 8, to the last of seven bytes in bit number 7 (64 decimal value).
Using this encoding, a group of 7 bytes from the .ALL format is transformed into a group of 8 SysEx bytes.

The length byte in each data packet indicates how many of those byte groups there are in the current data packet. Data is sent per 15 byte groups., resulting in a 127 byte SysEx packet, with the last data packet containing the remaining 6 byte groups. There's only five bytes in the .ALL format to fill the last byte group of the last data packet, and that byte group is padded with two FF(255) bytes.

The checksum byte is calculated as the XOR of all other bytes in the SysEx data packet, excluding the 240 and 247 start and stop bytes. When receiving a SysEx dump, the total XOR checksum of the bytes between 240 and 247 should therefore always be 0. (NB this is substantially different from the Roland way of doing SysEx checksums).

With this knowledge, I wrote a Perl script to convert .ALL files to SysEx (known as .syx) bytestreams. Owners of GEM WS1/WS2/WS400 keyboards who find themselves without floppies or without a working floppy drive can now load their .ALL files via a computer (with e.g. MIDI-OX or SysEx Librarian). If interested, send me an e-mail!