If you have ever streamed video from an ESP32, you have most likely used the CameraWebServer example: the camera delivers JPEG images, which the web server pushes one after the other to the browser. This works, but there is no sound, the bandwidth is high, and the result is not a real video stream that you could record or play in VLC. A real stream needs a video codec such as H.264, an audio codec Read more
Classic embedded text-to-speech (formant synthesizers like SAM or eSpeak) is tiny and fast, but it sounds like a robot from the 80s. Neural TTS sounds natural, but it is usually something you run on a PC or in the cloud. I wanted to know: what is the cheapest hardware that can run a neural text-to-speech model locally, faster than real time? To find out I ported tiny-tts to C++ as TinyTTS. It is a VITS-style, Read more
Why a tiny FPGA? I spend most of my time making microcontrollers do audio, graphics and machine learning, and I keep hitting the same wall: the CPU is too slow for the heavy inner loop. So I bought a cheap Sipeed Tang Nano 20K to see how far a hobby FPGA can push past that wall. Four projects later, the answer is: surprisingly far. The board is about the size of a stick of gum Read more
To produce the most natural-sounding speech, modern Text-To-Speach (TTS) systems rely on neural networks. TinyTTS is a proof of concept: It asks whether a VITS-style neural TTS model can run on a microcontroller at all. It is a header-only C++ port of tiny-tts. The model has about 1.6M parameters. It has four stages: text encoder, flow, duration predictor and a HiFi-GAN-style vocoder. There is no inference runtime such as TFLite Micro. Everything is hand-written C++ Read more
A follow-up to AudioTools video support, which covers container/codec selection (AVI vs. MP4 vs. MPG, MJPEG vs. H.264). This post is the other half: the pipeline that actually plays a file, a minimal sketch, and every class involved. Why this exists On a desktop, “play this video” hands the problem to an OS: a media framework demuxes the container, decodes on a scheduler-managed thread, and a compositor times frames to the display’s refresh. An ESP32 Read more
I added video support to my AudioTools library: Different container formats can be used to store video on audio together in the same file. Here is a quick overivew of the provided functionality that is relevant for Microcontrollers: Container Formats The simplest and therefore most recommended container format is AVI (Audio Video Interleave). It is simple to both encode and decode and you can easily mix audio and video together. It supports both different audio Read more
Material Design is an open-source design system created by Google in 2014 to help developers and designers build consistent, high-quality digital interfaces across Android, iOS, the web, and other platforms. I did not find any existing Arduino Library which would allow me to easily implement some Material Design Applications on Arduino, so I decided to implement my own TinyMaterialDesign library. I also added a comprehensive tutorial which explains the library in detail. Further information can Read more
I started to work on extending the AudioTools library to support different video formats. There are quite a few boards with LDC display that have a built in support for audio output. The cool thing is that they also provide an SD drive, so that you have plenty of room to store audio and videos. In a first step, I tested the audio and the LCD output. Here is an overview of the boards that Read more
In one of my last blogs, I gave an overview of the USB Class 2 Audio functionality provided by the AudioTools library Many users were struggeling to get the process of receiving data via USB and writing to I2S stable and working properly. In this blog I am providing a deep dive of the releated challenges and the solution approaches: Standard Processing Logic Here is a simple basic sketch that demonstrates the standard processing logic: Read more
The Fast Fourier Transform (FFT) is an efficient algorithm for computing the Discrete Fourier Transform (DFT): it converts a signal from the time domain into the frequency domain, revealing which frequencies are present in a signal and how strong they are. It is the workhorse behind spectrum analyzers, audio effects, filtering, and countless other DSP applications. I recently published a new FFT library that supports both integer and floating-point data types for its calculations. Using Read more