The Synthesis ToolKit (STK) Library for the Arduino ESP32 – An Introduction

I made the Synthesis ToolKit (SKT) available as Arduino Library.

The Synthesis ToolKit in C++ (STK) is a set of open source audio signal processing and algorithmic synthesis classes written in the C++ programming language. STK was designed to facilitate rapid development of music synthesis and audio processing software, with an emphasis on cross-platform functionality, realtime control, ease of use, and educational example code.

The information below is based on the original STK tutorial but was adapted to cover my extensions for the Arduino environment.

Introduction

Audio and control signals throughout STK use a floating-point data type, StkFloat, the exact precision of which can be controlled via a typedef statement in Stk.h. By default, an StkFloat is a double-precision floating-point value. Thus, the ToolKit can use any normalization scheme desired. The base instruments and algorithms are implemented with a general audio sample dynamic maximum of +/-1.0.

In general, the computation and/or passing of values is performed on a “single-sample” basis. For example, the stk::Noise class outputs random floating-point numbers in the range +/-1.0. The computation of such values occurs in the stk::Noise::tick() function. The following program will generate 20 random floating-point (StkFloat) values in the range -1.0 to +1.0:

#include "Noise.h"
using namespace stk;

void setup() {
  Serial.begin(115200);
  StkFloat output;
  Noise noise;
  for ( unsigned int i=0; i<20; i++ ) {
    output = noise.tick();
    Serial.println(output);
  }
}

void loop() {
}

Here is the result in the Arduino Serial Plotter:

Nearly all STK classes implement tick() functions that take and/or return sample values. Within the tick() function, the fundamental sample calculations are performed for a given class. Most STK classes consume/generate a single sample per operation and their tick() method takes/returns each sample “by value”. In addition, every class implementing a tick() function also provides one or more overloaded tick() functions that can be used for vectorized computations, as shown in the next example.

#include "Noise.h"
using namespace stk;

Noise noise;
StkFrames output(20, 1);   // initialize StkFrames to 20 frames and 1 channel (default: interleaved)

void setup() {
  Serial.begin(115200);
}

void loop() {
  noise.tick( output );
  for ( unsigned int i=0; i<output.size(); i++ ) {
    Serial.println(output[i]);
  }
}

The generated output is the same as in the first example.

Hello Sine! (Input to Output)

We’ll continue our introduction to the Synthesis ToolKit with a simple sine-wave oscillator program. STK provides two different classes for sine-wave generation. In STK it is quite common to send the output to a file. In Arduino however we rarely have any file system available but we need usually write output to serial. Here is how it can be done:

#include "SineWave.h"
#include "ArdStreamOut.h"

using namespace stk;

SineWave input;
ArdStreamOut output(Serial);

void setup() {
  Serial.begin(115200);

  Stk::setSampleRate( 44100.0 );
  input.setFrequency( 440.0 );
}

void loop() {
  output.tick( input.tick() );
}

We just copy the input to the output which is Serial in this case. And the output in the Arduino Plotter looks as follows:

The ArdStreamOut is expecting a stream. This is quite powerful if you consider that Arduino provides different implementations of Streams!

The following Arduino specific classes are provided:

• ArdStreamOut (Output as string to Stream)
• ArdI2SOut (Output to I2S pins or internal DAC)
• ArdStreamBinaryOut (Output as binary data to Stream)
• ArdStreamHexOut (Output as data converted to hex to Stream)

Sending data over the network

The example from above can easily be extended to send the data over the network instead of just to the serial port:

#include "SineWave.h"
#include "ArdStreamOut.h"
#include "WiFi.h" // ESP32 WiFi include
#include "ArdUdp.h" 

using namespace stk;

SineWave input;
IPAddress ip(192, 168, 1, 35);
ArdUdp udp(ip, 7000);
ArdStreamBinaryOut output(udp);
const char *SSID = "your ssid";
const char *PWD = "your password";


void setup() {
  Serial.begin(115200);
  WiFi.begin(SSID, PWD);
  while (WiFi.status() != WL_CONNECTED) {
    Serial.print('.');
  }

  Serial.print("Connected to IP address: ");
  Serial.println(WiFi.localIP());

  Stk::setSampleRate( 44100.0 );
  input.setFrequency( 440.0 );
}

void loop() {
  output.tick( input.tick() );
}

I am using here ArdStreamBinaryOut because I would like to send the data as is and not as text. We provide the ArdUdp class which is just a simple wrapper over the Arduino WiFiUDP class. It is sending all output to the same destination – which is indicated in the constructor. As far as TCP is concerned we can directly use the standard Arduino classes

Instruments

The ToolKit comes with a wide variety of synthesis algorithms, all of which inherit from the stk::Instrmnt class. In this example, we’ll fire up an instance of the stk::Clarinet synthesis class and show how its frequency can be modified over time. The instruments are all implementing the noteOn() and noteOff() methods. The noteOff() method need to be explicitly called on the Instruments where the tone is not decaying over time.

#include "ArdStreamOut.h"
#include "ArdMidiCommon.h"
#include "Clarinet.h"

using namespace stk;

ArdStreamOut output(Serial);
Clarinet instrument(440);
int note = 90; // starting midi note

void setup() {
  Serial.begin(115200);
}

void loop() {
  note += rand() % 10 - 5; // generate some random offset
  float frequency = ArdMidiCommon::noteToFrequency(note);

  instrument.noteOn( frequency, 0.5 );
  long endTime = millis()+1000;
  while (millis()<endTime){
      output.tick( instrument.tick() );
  }
  delay( 100 );

}

That’s not too difficult to get an instrument to play. It would be quite natural to link the instrument up to process incoming Midi Messages. But we get to this later on.

Unfortunately quite a few instruments are using sampled raw files as input. I am still working on a work-around for this. Here is the class overview of all instruments

              .- FM - (HevyMetl, PercFlut, Rhodey, Wurley, TubeBell, BeeThree, FMVoices)
              |
              |- Modal - ModalBar
              |
              |- VoicForm
              |
              |- Sampler - Moog
              |
              |- Resonate
              |
              |- Mandolin
  - Instrmnt -|
              |- Drummer
              |
              |- Clarinet, BlowHole, Saxofony, Flute, Brass, BlowBotl,
              |  Bowed, Plucked, StifKarp, Sitar, Recorder
              |
              |- Shakers
              |
              |- BandedWG
              |
              |- Mesh2D
              |
              .- Whistle

Sending the output of Instruments to Bluetooth

I was starting to get into the STK framework when I was looking to implement some ways to generate sound for my Bluetooth A2DP Sink.

I have added this functionality as well:

#include "ArdBtSource.h"
#include "ArdMidiCommon.h"
#include "Clarinet.h"
#include "Voicer.h"

using namespace stk;

Clarinet clarinet(440);
Voicer voicer;
ArdBtSource bt;
StkFloat note = 64; // 0 to 128
StkFloat amplitude = 100; // 0 to 128

void setup() {
  Serial.begin(115200);
  Stk::setSampleRate( 44100.0 );
  voicer.addInstrument(&clarinet);

  bt.start("RadioPlayer", voicer);
}

void loop() {
  if (bt.isConnected()) {
    Serial.print("playing ");
    Serial.println(++note);

    voicer.noteOn( note, amplitude );
    delay(900);
    voicer.noteOff( note, 20 );
    delay(200);
    if (note>=90) {
      note = 30;
    }
  }
}

The Voicer is basically just a simple way to play multiple instruments at the same time. The Bluetooth functionality is made available by the ArdBtSource class. The start command launches a connection to the “RadioPlayer” Bluetooth sink (e.g speakers).

Please note that unlike the instruments which expect a frequency, the Voicer is expecting a Midi note as input!

And this introduces us to the next topic. I am about to add some Midi functionality to the framework. But this will be the topic in one of my next posts. In the meantime if you are curious you can have a look at the class documentation provided by the current framework.

Conclusion

The ESP32 provides enough power do generate music and with it’s WIFI and Bluetooth functionality it can be the basis for some very interesting music projects.