Skip to main content

GEM WS2 MIDI System Exclusive structure and checksums

MIDI is the standard for communication between electronic music instruments like keyboards and synthesizers. And computers! While tinkering with an old floppy-less GEM WS2 keyboard, I wanted to figure out the structure of their System Exclusive memory dumps. SysEx is the vendor-specific (and non-standard) part of MIDI. Vendors can use it for real-time instructions (changing a sound parameter in real-time) and for non-real-time instructions (sending or loading a configuration, sample set, etc.).

In the GEM WS2, there's two ways of saving the memory (voices, globals, styles and songs): in .ALL files on floppy, and via MIDI SysEx.

The .ALL files are binary files, 60415 bytes long. The only recognizable parts are the ASCII encoded voice and global names. The SysEx dumps are 73691 bytes long. As always in MIDI, only command start (and end) bytes have MSB 1, and all data bytes have MSB 0. The data is spread out over 576 SysEx packets, preceded by one SysEx packet with header information.

Each SysEx data packet starts with these bytes (decimal representation):

  • 240 (SysEx start) 
  • 47 (GeneralMusic / GEM / Elka manufacturer ID) 
  • 2 (the header packet has a 1 here, the data packes have a 2) 
  • a six-bit packet counter (data packet number MOD 64)
  • 15 (data length, discussed below)
  • then there's room for 120 data bytes
  • one checksum byte (discussed below)
  • 247 (SysEx end)


Because the original data (the WS2 memory and the .ALL file) has 8 bits per byte, and MIDI SysEx bytes can only have 7 bits (MSB 0), GEM uses an encoding to go from one to the other:
Seven 8-bit bytes have their LSB stripped, and the LSB's form byte number 8, from the first of seven bytes in the LSB of byte number 8, to the last of seven bytes in bit number 7 (64 decimal value).
Using this encoding, a group of 7 bytes from the .ALL format is transformed into a group of 8 SysEx bytes.

The length byte in each data packet indicates how many of those byte groups there are in the current data packet. Data is sent per 15 byte groups., resulting in a 127 byte SysEx packet, with the last data packet containing the remaining 6 byte groups. There's only five bytes in the .ALL format to fill the last byte group of the last data packet, and that byte group is padded with two FF(255) bytes.

The checksum byte is calculated as the XOR of all other bytes in the SysEx data packet, excluding the 240 and 247 start and stop bytes. When receiving a SysEx dump, the total XOR checksum of the bytes between 240 and 247 should therefore always be 0. (NB this is substantially different from the Roland way of doing SysEx checksums).

With this knowledge, I wrote a Perl script to convert .ALL files to SysEx (known as .syx) bytestreams. Owners of GEM WS1/WS2/WS400 keyboards who find themselves without floppies or without a working floppy drive can now load their .ALL files via a computer (with e.g. MIDI-OX or SysEx Librarian). If interested, send me an e-mail!




Comments

Ramon31177 said…
Hoi Bert,

Ik kwam je blog tegen en hopelijk kun jij me verder helpen. Ik heb een GEM WS1, maar mijn diskdrive is er mee gestopt... Nou ja, gestopt, bij elke diskette heeft hij aan dat de data corrupt is. Ik ben dus op zoek naar een manier om mijn .ALL files te kunnen gebruiken, zodat ik alles via MIDI kan gaan overzetten. Het klinkt als hetgeen je beschrijft mij zou kunnen helpen... Ik hoor je graag!

groet,
Ramon
Bert de Bruijn said…
Ramon, ik heb je een Google Hangouts bericht gestuurd. Heb je dat gekregen?
אסף ס said…
HAY BERT
I HAVE GEM WS2
I MOVED THEM FROM THE FLPOPPY TO MY PC ALL THE FILES ARE WITH EXTENSION .ALL
I WANT TO CONVERT THEM TO MIDI
HOW CAN I DO THAT ?
i cant find your email to write to you
can you write here your email
or email me :asaf0968@gmail.com
THANKS
Bert de Bruijn said…
Hi, I can share the script with you, but it's easier to send (wetransfer, google drive, ...) the .ALL files to me, I'll convert them and send them back. I did that for Ramon as well. I'll send you a message with my e-mail.
bello said…
Hi Bert,
I have GEM WS2, good old machine.
I need to convert some .all files, can You semd me your mail please?
bello@bih.net.ba
Thanks.
Hasan Arikan said…
Hi
I have GEM WS2
I need to convert some .all files, can You semd me your mail
my email :osmanli1876@hotmail.com.tr

Thanks.
Gunter Van Deun said…
Hello there,

I plan to start again with my WS2 GEM device, and was wondering if we can share files with eachother ?
I have some music files (.all) i can also share with you. Of course I still will need to find a way to get the from and later on the floppy disk.....

Thanks !
Bert de Bruijn said…
Hi Gunter, send me a mail and I'll share the .all files I have with you. Do you have your WS2 connected to a computer via MIDI ? If you have the midi save/load option in the menu, then transferring system exclusive dumps may be easier. You can do the load from/save to floppy on your WS2 then, without a floppy drive on your PC (which is also becoming rare). My gmail address is bertdb.
John 177 said…
Hi Bert. Maybe you can help?

I have a GEM WS2 KEYBOARD workstation. I can't find the Midi save/load option in the Midi menu on this unit or does this only apply to the WS2 MIDI workstation? My WS2 has a working floppy drive but many of my floppy disks are corrupt and i would like to re-format them and in turn reload them with the .ALL files on my PC which i have downloaded to my HD from the internet. Unfortunately my pc has no FD drive installed so a Midi dump option using Midi-Ox would've been great! Another possible option is to install a USB FD Emulator in my WS2 that will emulate a 720kb drive? Any advice is welcome Bert!

Many thanks in advance,

John
Bert de Bruijn said…
Hi John 177. The MIDI save/load options were added in an updated firmware, which the early WS2's didn't have factory-installed. There were other improvements like a programmable drumkit. The newer WS2's are sometimes referred to as WS2ii, and as far as I know the newer firmware is the only difference. The early WS2 my parents once bought was upgraded by means of a chip replacement by the music shop where they bought the keyboard. When you power up the keyboard, you should see V1.10 as the software version for an upgraded WS2. I think the WS2 midi workstation was introduced after the WS2 and WS400 keyboards, so it may have never existed with the original firmware. I have never owned a WS2 midi workstation. I have looked at the Floppy emulator option a bit before writing my .ALL converter, but never found any that I could confirm as compatible with the non-standard DB25 (IIRC) floppy connector used in the WS2 without actually buying one. My second-hand WS2 has the 1.10 firmware with the MIDI save/load option, so I ended up experimenting with that instead. (All of this is based on what I remember and found. There is no official changelog of these devices that I know of).

Popular posts from this blog

How to solve "user locked out due to failed logins" in vSphere vMA

In vSphere 6, if the vi-admin account get locked because of too many failed logins, and you don't have the root password of the appliance, you can reset the account(s) using these steps:

reboot the vMAfrom GRUB, "e"dit the entry"a"ppend init=/bin/bash"b"oot# pam_tally2 --user=vi-admin --reset# passwd vi-admin # Optional. Only if you want to change the password for vi-admin.# exitreset the vMAlog in with vi-admin These steps can be repeated for root or any other account that gets locked out.

If you do have root or vi-admin access, "sudo pam_tally2 --user=mylockeduser --reset" would do it, no reboot required.

Volkswagen UHV bluetooth touch adapter & its problems

My Volkswagen car has the "universal cellphone preparation" UHV built-in. This is the main part of a car kit, but requires an additional adapter for connecting to a cellphone. At first, I was using an adapter for my good old Nokia 6310, even after I changed to the Nokia E71. Connecting was easy: pair the phone with the "VW UHV" bluetooth entity, and done. This has the phone connected to the car kit at all times, so even non-call-related functions use the car audio system (e.g. voice recognition).
But progress will have its way, no matter what happens. So in comes the "bluetooth touch adapter". Instead of a phone-specific adapter, this is a small touchscreen device that slots into the UHV dashboard mount. Connecting a phone is very different now:
the Bluetooth Touch Adapter connects to the "VW UHV" device via bluetooth
the phone connects to "Touch Adapter" device, also via bluetoothThe device doesn't allow step 2 if step 1 didn't s…

Multiple VLANs on a Synology NAS

Synology, like other SOHO/SMB NAS vendors, touts VLAN functionality with their current DSM 4.1 software. However, the web interface just lets you specify one VLAN tag to use over each eth interface (or bond interface).
Manual approachIn the busybox environment that you can ssh into as root (after enabling ssh through the webinterface), there's all the tools you need to use multiple VLANs over one link (eth or bond), however:
First you insert the 802.1q module into the Linux kernel:
 /sbin/lsmod | /bin/grep -q 8021q || /sbin/insmod /lib/modules/8021q.koThen you add each VLAN you need to every interface (bond0 in this example)
 /sbin/vconfig add bond0 4And finally you can configure IP addresses on every interface.vlan combination (bond0.4 in this example)
 /sbin/ifconfig bond0.4 192.168.4.1 broadcast 192.168.4.255 netmask 255.255.255.0The same type of script would work on a QNAP NAS too, by the way. They offer 8021q.ko and vconfig in their commandline environment as well.
Packets from…