Skip to main content

GEM WS2 MIDI System Exclusive structure and checksums

MIDI is the standard for communication between electronic music instruments like keyboards and synthesizers. And computers! While tinkering with an old floppy-less GEM WS2 keyboard, I wanted to figure out the structure of their System Exclusive memory dumps. SysEx is the vendor-specific (and non-standard) part of MIDI. Vendors can use it for real-time instructions (changing a sound parameter in real-time) and for non-real-time instructions (sending or loading a configuration, sample set, etc.).

In the GEM WS2, there's two ways of saving the memory (voices, globals, styles and songs): in .ALL files on floppy, and via MIDI SysEx.

The .ALL files are binary files, 60415 bytes long. The only recognizable parts are the ASCII encoded voice and global names. The SysEx dumps are 73691 bytes long. As always in MIDI, only command start (and end) bytes have MSB 1, and all data bytes have MSB 0. The data is spread out over 576 SysEx packets, preceded by one SysEx packet with header information.

Each SysEx data packet starts with these bytes (decimal representation):

  • 240 (SysEx start) 
  • 47 (GeneralMusic / GEM / Elka manufacturer ID) 
  • 2 (the header packet has a 1 here, the data packes have a 2) 
  • a six-bit packet counter (data packet number MOD 64)
  • 15 (data length, discussed below)
  • then there's room for 120 data bytes
  • one checksum byte (discussed below)
  • 247 (SysEx end)


Because the original data (the WS2 memory and the .ALL file) has 8 bits per byte, and MIDI SysEx bytes can only have 7 bits (MSB 0), GEM uses an encoding to go from one to the other:
Seven 8-bit bytes have their LSB stripped, and the LSB's form byte number 8, from the first of seven bytes in the LSB of byte number 8, to the last of seven bytes in bit number 7 (64 decimal value).
Using this encoding, a group of 7 bytes from the .ALL format is transformed into a group of 8 SysEx bytes.

The length byte in each data packet indicates how many of those byte groups there are in the current data packet. Data is sent per 15 byte groups., resulting in a 127 byte SysEx packet, with the last data packet containing the remaining 6 byte groups. There's only five bytes in the .ALL format to fill the last byte group of the last data packet, and that byte group is padded with two FF(255) bytes.

The checksum byte is calculated as the XOR of all other bytes in the SysEx data packet, excluding the 240 and 247 start and stop bytes. When receiving a SysEx dump, the total XOR checksum of the bytes between 240 and 247 should therefore always be 0. (NB this is substantially different from the Roland way of doing SysEx checksums).

With this knowledge, I wrote a Perl script to convert .ALL files to SysEx (known as .syx) bytestreams. Owners of GEM WS1/WS2/WS400 keyboards who find themselves without floppies or without a working floppy drive can now load their .ALL files via a computer (with e.g. MIDI-OX or SysEx Librarian). If interested, send me an e-mail!




Comments

Unknown said…
Hoi Bert,

Ik kwam je blog tegen en hopelijk kun jij me verder helpen. Ik heb een GEM WS1, maar mijn diskdrive is er mee gestopt... Nou ja, gestopt, bij elke diskette heeft hij aan dat de data corrupt is. Ik ben dus op zoek naar een manier om mijn .ALL files te kunnen gebruiken, zodat ik alles via MIDI kan gaan overzetten. Het klinkt als hetgeen je beschrijft mij zou kunnen helpen... Ik hoor je graag!

groet,
Ramon
Bert de Bruijn said…
Ramon, ik heb je een Google Hangouts bericht gestuurd. Heb je dat gekregen?
Unknown said…
HAY BERT
I HAVE GEM WS2
I MOVED THEM FROM THE FLPOPPY TO MY PC ALL THE FILES ARE WITH EXTENSION .ALL
I WANT TO CONVERT THEM TO MIDI
HOW CAN I DO THAT ?
i cant find your email to write to you
can you write here your email
or email me :asaf0968@gmail.com
THANKS
Bert de Bruijn said…
Hi, I can share the script with you, but it's easier to send (wetransfer, google drive, ...) the .ALL files to me, I'll convert them and send them back. I did that for Ramon as well. I'll send you a message with my e-mail.
bello said…
Hi Bert,
I have GEM WS2, good old machine.
I need to convert some .all files, can You semd me your mail please?
bello@bih.net.ba
Thanks.
Unknown said…
Hi
I have GEM WS2
I need to convert some .all files, can You semd me your mail
my email :osmanli1876@hotmail.com.tr

Thanks.
Unknown said…
Hello there,

I plan to start again with my WS2 GEM device, and was wondering if we can share files with eachother ?
I have some music files (.all) i can also share with you. Of course I still will need to find a way to get the from and later on the floppy disk.....

Thanks !
Bert de Bruijn said…
Hi Gunter, send me a mail and I'll share the .all files I have with you. Do you have your WS2 connected to a computer via MIDI ? If you have the midi save/load option in the menu, then transferring system exclusive dumps may be easier. You can do the load from/save to floppy on your WS2 then, without a floppy drive on your PC (which is also becoming rare). My gmail address is bertdb.
Unknown said…
Hi Bert. Maybe you can help?

I have a GEM WS2 KEYBOARD workstation. I can't find the Midi save/load option in the Midi menu on this unit or does this only apply to the WS2 MIDI workstation? My WS2 has a working floppy drive but many of my floppy disks are corrupt and i would like to re-format them and in turn reload them with the .ALL files on my PC which i have downloaded to my HD from the internet. Unfortunately my pc has no FD drive installed so a Midi dump option using Midi-Ox would've been great! Another possible option is to install a USB FD Emulator in my WS2 that will emulate a 720kb drive? Any advice is welcome Bert!

Many thanks in advance,

John
Bert de Bruijn said…
Hi John 177. The MIDI save/load options were added in an updated firmware, which the early WS2's didn't have factory-installed. There were other improvements like a programmable drumkit. The newer WS2's are sometimes referred to as WS2ii, and as far as I know the newer firmware is the only difference. The early WS2 my parents once bought was upgraded by means of a chip replacement by the music shop where they bought the keyboard. When you power up the keyboard, you should see V1.10 as the software version for an upgraded WS2. I think the WS2 midi workstation was introduced after the WS2 and WS400 keyboards, so it may have never existed with the original firmware. I have never owned a WS2 midi workstation. I have looked at the Floppy emulator option a bit before writing my .ALL converter, but never found any that I could confirm as compatible with the non-standard DB25 (IIRC) floppy connector used in the WS2 without actually buying one. My second-hand WS2 has the 1.10 firmware with the MIDI save/load option, so I ended up experimenting with that instead. (All of this is based on what I remember and found. There is no official changelog of these devices that I know of).
Bert de Bruijn said…
For those who want to experiment with the conversion themselves, I've shared my Perl script on github: https://github.com/bertdb/gem-ws2-floppy-to-midi-dump/ . I'm publishing it under the GPL license, so you're allowed to use the program freely, but you're not allowed to change and distribute it unless you share the source code.

If you share your .ALL files with me, I can include them in the collection of over 150 .ALL files I have built up. For example, nine of the original GEM WS2 floppy series have been found, but many are still missing.
Unknown said…

Hello
Bert de Bruijn
I need your help !!!
1) I have a Gem WS2 II keyboard with floppy disk.
I want to convert my .all files to midi with your Perl script.
When I run ws2-transform.pl I get this message:
C: \> ws2-transform.pl
Use of uninitialized value $ syxfilename in substitution (s ///) at C: \ ws2-transform.pl line 63.
Use of uninitialized value $ allfilename in string eq at C: \ ws2-transform.pl line 67.
Use of uninitialized value $ syxfilename in string eq at C: \ ws2-transform.pl line 67.
Use of uninitialized value $ allfilename in concatenation (.) Or string at C: \ ws2-transform.pl line 67.
Use of uninitialized value $ allfilename in concatenation (.) Or string at C: \ ws2-transform.pl line 77.
Use of uninitialized value $ allfilename in concatenation (.) Or string at C: \ ws2-transform.pl line 77.
Could not open for reading at C: \ ws2-transform.pl line 77.
How can I solve the problem ?
2) How do I enable sysex to transfer midi files with the load midi command and the midi-ox program?
3) I have about 30 songs of song, global and style that I could exchange with yours if you wish.
My email: meschinimauro2@gmail.com
Bert de Bruijn said…
Mauro, you should provide a filename after ws2-transform.pl . I hope the script will work on Windows, I only tested it on Linux.

Letting the WS2 wait for .syx data from MIDI-OX is in the midi menu, called "LOAD MIDI" if I remember correctly (page 5 or 6 I believe, can't check because I don't have a WS2 anymore). You need the WS2ii firmware, in the original WS2 1.00 firmware this function was not present.
Gary Strickland said…
I have a lot of floppys with "ALL" files from my old Bachmann WS2 keyboard I'd like to convert to MIDI. Is there any way you could help? I would be so grateful. Also, I no longer have the WS2 keyboard which is why I'd like to convert the files so I can use them on my newer Korg PA series keyboard.
Thank you so much for any input,
Gary Strickland
Bert de Bruijn said…
Hello Gary. Unfortunately, the only conversion I can do is from the floppy based memory dump to the MIDI based memory dump format that the WS2 understands. So the converted MIDI SysEx dumps I can create from your .ALL files will only load on a WS2 keyboard or module. If you know someone who still has a WS2, you can always load them there, and record songs or patterns via MIDI OUT, or as audio.
Ivano Cirillo said…
hy and tanx for this... sorry for my english. i have a lot of .all files i have saved on my pc from my floppy. how can i export in midi files? i have download this https://github.com/bertdb/gem-ws2-floppy-to-midi-dump/
but i dont have understand ho i can do. please send me a mail this files are a remember of my father and i want to take care.

tanx a lot can you write at cirillo.ivano@gmail.com

best regards
maybe a little off-topic, but has anybody information about recognized system exclusive messages for the GEM SX2/SX3?
Bert de Bruijn said…
Leo, I never owned a GEM SX2/3. I can't help you with that. It wouldn't surprise me if they used the same chopped-LSB 8-to-7-bit encoding when sending raw memory data over MIDI.
ok, thanks Bert, just was looking how to change Styles and other parameters etc via sys exclusives because my SX3 has several defective buttons, in fact beyond repair but i am kind a fond of the instrument and have several nice sounds and styles.
Found already that i can change Styles and performances via Controller 70 on the common channel. This leaves me with some of the other functions which i cannot change via MIDI.

best regards
Leo
Bert de Bruijn said…
Leo, the WS2 has a MIDI channel for "control", by default channel 16. It sends regular (non-SysEx) control messages for each button you push. Quick way to check that: connect a MIDI cable from out to in, and use the buttons. If e.g. you step through menu's two pages at a time, this means the button is working, and the message it sends is also being received, resulting in a second trigger. If this works on your WX2/WX3, then you can start testing which control message triggers which function.
Bert, great suggestion, also the SX3 has a channel for control, this is where i can select Styles and Performances via Bank Select followed by Program changes.
Never looked at the data it sends for keypresses because it is not written in the MIDi-implementation chart, but will try this soon and let you know.


Thanks
Bert de Bruijn said…
This comment has been removed by the author.
Bert de Bruijn said…
Patrick, you can send me a mail via bert+ws2 at debruijn.be

Popular posts from this blog

Volkswagen UHV bluetooth touch adapter & its problems

My Volkswagen car has the "universal cellphone preparation" UHV built-in. This is the main part of a car kit, but requires an additional adapter for connecting to a cellphone. At first, I was using an adapter for my good old Nokia 6310, even after I changed to the Nokia E71. Connecting was easy: pair the phone with the "VW UHV" bluetooth entity, and done. This has the phone connected to the car kit at all times, so even non-call-related functions use the car audio system (e.g. voice recognition). But progress will have its way, no matter what happens. So in comes the "bluetooth touch adapter". Instead of a phone-specific adapter, this is a small touchscreen device that slots into the UHV dashboard mount. Connecting a phone is very different now: the Bluetooth Touch Adapter connects to the "VW UHV" device via bluetooth the phone connects to "Touch Adapter" device, also via bluetooth The device doesn't allow step 2 if step 1 didn'

Reset lost root password on vSphere ESXi 6.7

VMware's solution to a lost or forgotten root password for ESXi is simple: go to  https://kb.vmware.com/s/article/1317898?lang=en_US  and you'll find that "Reinstalling the ESXi host is the only supported way to reset a password on ESXi". If your host is still connected to vCenter, you may be able to use Host Profiles to reset the root password, or alternatively you can join ESXi in Active Directory via vCenter, and log in with a user in the "ESX Admins" AD group. If your host is no longer connected to vCenter, those options are closed. Can you avoid reinstallation? Fortunately, you can. You will need to reset and reboot your ESXi though. If you're ready for an unsupported deep dive into the bowels of ESXi, follow these steps: Create a bootable Linux USB-drive (or something else you can boot your server with). I used a CentOS 7 installation USB-drive that I could use to boot into rescue mode. Reset your ESXi and boot from the Linux medium. Ident