The Synthesis ToolKit (STK) Library for the Arduino ESP32 – Supporting MIDI and BLE

I made the Synthesis ToolKit (SKT) available as Arduino Library.

The Synthesis ToolKit in C++ (STK) is a set of open source audio signal processing and algorithmic synthesis classes written in the C++ programming language. STK was designed to facilitate rapid development of music synthesis and audio processing software, with an emphasis on cross-platform functionality, realtime control, ease of use, and educational example code.

Standard STK uses Skini instead of MIDI. This is a simple text based format which provides the same functionality as MIDI.

However, my intention was to use a simple MIDI keyboard as input device to generate music on an ESP32. In my initial use case I just wanted to connect the device via Bluetooth Low Energy (BLE).

After having implemented a BLE server that was able to receive MIDI messages (which was the difficult part) I decided to add some sending functionality as well (which was pretty easy) – and having all this functionality it was no effort at all to refactor the code a little bit to provide additional communication possibilities.

I have integrated the MIDI functionality into the library:

Sending MIDI Messages

Midi messages can be sent by using the following methods on an output device

• void noteOn(uint8_t note, uint8_t velocity, int8_t channel=-1);
• void noteOff(uint8_t note, uint8_t velocity, int8_t channel=-1);
• void pitchBend(uint16_t value, int8_t channel=-1);
• void channelPressure(uint8_t value, int8_t channel=-1);
• void polyPressure(uint8_t valuePar, int8_t channel=-1);
• void programChange(uint8_t program, int8_t channel=-1);
• void allNotesOff( int8_t channel=-1);
• void resetAllControllers( int8_t channel=-1);
• void localControl( bool active, int8_t channel=-1);
• void controlChange(uint8_t msg, uint8_t value, int8_t channel=-1);

Output MIDI Messages to Serial

Output of MIDI messages are done with the help of the ArdMidiStreamOut class. It expects a Stream as argument, which makes it so flexible.

Here is the implementation that sends the output to Serial. MIDI is a binary protocol, so you will not see anything meaningful in your Serial Monitor:

#include "ArdMidiStreamOut.h"

using namespace stk;

ArdMidiStreamOut out(Serial);
StkFloat note = 64; // 0 to 128
StkFloat amplitude = 100; // 0 to 128

void setup() {
    Serial.begin(115200);
}

void loop() {
    Serial.println();
    Serial.print("playing ");
    Serial.println(++note);

    out.noteOn( note, amplitude );
    delay(900);
    out.noteOff( note, 20 );
    delay(200);
    if (note>=90) {
      note = 30;
    }
}

Output MIDI Messages to IP

In order to do communication with TCP/IP you can use the standard Arduino WiFiClient class. Of cause you need to have a listening server on the receiving side for this to work:

#include <WiFi.h>
#include <WiFiMulti.h>
#include "ArdMidiStreamOut.h"

using namespace stk;

IPAddress ip(192, 168, 1, 35);
int port = 9999;
WiFiClient client;
ArdMidiStreamOut out(client);
const char *SSID = "your ssid";
const char *PWD = "your password";

StkFloat note = 64; // 0 to 128
StkFloat amplitude = 100; // 0 to 128

void setup() {
  Serial.begin(115200);
  WiFi.begin(SSID, PWD);
  while (WiFi.status() != WL_CONNECTED) {
    Serial.print('.');
    delay(1000);
  }

  Serial.print("Connected to IP address: ");
  Serial.println(WiFi.localIP());

  client.connect(ip, port);
}

void loop() {
    Serial.print("playing ");
    Serial.println(++note);

    out.noteOn( note, amplitude );
    delay(900);
    out.noteOff( note, 20 );
    delay(200);
    if (note>=90) {
      note = 30;
    }
}

We need to login into WIFI and then connect to the requested IP address and port. This is just plain vanilla Arduino code. Since we have given the WifiClient as argument to the ArdMidiStreamOut class, the information is automatically sent to the expected destination.

Output MIDI Messages to UDP

We have already seen an example in the introduction on how to send out sound via UDP. This is very similar and we use ArdMidiStreamOut instead of ArdStreamBinaryOut.

#include "WiFi.h" 
#include "ArdUdp.h" 
#include "ArdMidiStreamOut.h"

using namespace stk;

IPAddress ip(192, 168, 1, 35);
ArdUdp udp(ip, 7000);
ArdMidiStreamOut out(udp);
const char *SSID = "your ssid";
const char *PWD = "your password";

StkFloat note = 64; // 0 to 128
StkFloat amplitude = 100; // 0 to 128

void setup() {
  Serial.begin(115200);
  WiFi.begin(SSID, PWD);
  while (WiFi.status() != WL_CONNECTED) {
    Serial.print('.');
    delay(1000);
  }

  Serial.print("Connected to IP address: ");
  Serial.println(WiFi.localIP());
}

void loop() {
    Serial.print("playing ");
    Serial.println(++note);

    out.noteOn( note, amplitude );
    delay(900);
    out.noteOff( note, 20 );
    delay(200);
    if (note>=90) {
      note = 30;
    }
}

Receiving MIDI Messages

Receiving messages is easy as well. You use the class ArdMidiStreamIn together with a Stream and a ArdMidiEventHandler which implements the parsing of MIDI messages.

Input MIDI Messages from Serial

#include "Voicer.h"
#include "Clarinet.h"
#include "ArdMidiStreamIn.h"

using namespace stk;

Voicer voicer;
Clarinet clarinet(440);
ArdMidiEventHandler handler(&voicer);
ArdMidiStreamIn in(Serial, handler);

void setup() {
  Serial.begin(115200);
  voicer.addInstrument(&clarinet);
}

void loop() {
  in.loop();
}

In the setup method we tell the voicer to play the clarinet when we receive a corresponding command on channel 0.
In the loop we need to call the loop() method from the ArdMidiStreamIn to trigger the receiving and parsing of the MIDI messages.

Input MIDI Messages from UDP

We can use the same logic as above, but we provide the standard Arduino WiFiUDP to the ArdMidiStreamIn:

#include <WiFi.h>
#include <WiFiClient.h>
#include <WiFiUdp.h>
#include <ArdMidiStreamIn.h>
#include "Voicer.h"
#include "Clarinet.h"

using namespace stk;


WiFiUDP udp;
Clarinet clarinet(440);
Voicer voicer;
ArdMidiEventHandler handler(&voicer);
ArdMidiStreamIn in(udp, handler );
int localPort = 9000;

void setup() {
  Serial.begin(115200);
  WiFi.begin(SSID, PWD);
  while (WiFi.status() != WL_CONNECTED) {
    Serial.print('.');
    delay(1000);
  }

  Serial.print("Connected to IP address: ");
  Serial.println(WiFi.localIP());  

  voicer.addInstrument(&clarinet,0);
  udp.begin(localPort);
}

void loop() {
  in.loop();
}

BLE Server

The MIDI BLE Server functionality is provided by the ArdMidiBleServer class. Above you have already seen all relevant components and how they work together.

Sending of MIDI Messages over BLE

#include <ArdMidiBleServer.h>

using namespace stk;

ArdMidiBleServer ble("MidiServer");

StkFloat note = 64; // 0 to 128
StkFloat amplitude = 100; // 0 to 128

void setup() {
  Serial.begin(115200);
  ble.start();
}

void loop() {
  Serial.print("playing ");
  Serial.println(++note);

  ble.noteOn( note, amplitude );
  delay(900);
  ble.noteOff( note, 20 );
  delay(200);
  if (note>=90) {
    note = 30;
  }
}

Receiving of MIDI Messages over BLE

In order to receive an process inbound MIDI messages we use the ArdMidiBLEEventHandler class which expects a voicer.

#include <Voicer.h>
#include <Clarinet.h>
#include <ArdMidiBleEventHandler.h>
#include <ArdMidiBleServer.h>

using namespace stk;

Voicer voicer;
Clarinet clarinet(440);
ArdMidiBleEventHandler handler(&voicer);
ArdMidiBleServer ble("MidiServer", &handler);

void setup() {
  Serial.begin(115200);
  voicer.addInstrument(&clarinet, 0);
  ble.start(voicer);
}

void loop() {
}

Of cause you could add the loop processing from the sending example here as well and you would have a MIDI BLE server that can send and receive MIDI messages at the same time!

BLE Client

After having seen how to use the BLE Server I finally demonstrate the corresponding ArdMidiBleClient class which acts as BLE Client.
Like the server it can receive and send out MIDI messages at the same time:

#include <Voicer.h>
#include <Clarinet.h>
#include <ArdMidiBleEventHandler.h>
#include <ArdMidiBleServer.h>

using namespace stk;

Voicer voicer;
Clarinet clarinet(440);
ArdMidiBleEventHandler handler(&voicer);
ArdMidiBleServer ble("MidiServer", &handler);

void setup() {
  Serial.begin(115200);
  voicer.addInstrument(&clarinet, 0);
  ble.start(voicer);
}

StkFloat note = 64; // 0 to 128
StkFloat amplitude = 100; // 0 to 128

void loop() {
  Serial.print("playing ");
  Serial.println(++note);

  ble.noteOn( note, amplitude );
  delay(900);
  ble.noteOff( note, 20 );
  delay(200);
  if (note>=90) {
    note = 30;
  }
}

In this example we receive and process MIDI messages on channel 0 and send out messages on channel 1.

Conclusion

We can receive and process MIDI messages over different communication channels – including BLE. Combining this with the standard sound generation functionality of the STK framework we can now build some powerful Music Projects quite easily!

And if you discover that one of the provided Classes does not quite meet your exact requirements: This is C++ – you can just create a Subclass and overwrite the method that you want to extend…