An open-source project that enables real-time English speech recognition on an Espressif ESP32-P4 board without an internet connection, cloud services or a separate AI accelerator.
The project turns an ESP32 development board into an offline voice assistant that converts spoken words into text and responds to voice commands. Since all the processing takes place on the board, audio is not sent to an external server. The system also continues to work without Wi-Fi, making it suitable for applications in which connectivity is unreliable and privacy is important.
The project called esp32-pie-asr. It was developed by Boumedine Billal and is published on GitHub. It runs on the Espressif ESP32-P4X rev. v3, a dual-core RISC-V microcontroller. It has two processor cores and supports PIE vector instructions instead of relying on a dedicated neural accelerator. The project also provides live speech captions on a connected display, along with six voice-control demonstrations.
At its core is Kroko, an open-source speech recognition model from Banafo. It is a 65.7-million-parameter Zipformer 2 Transducer neural network that processes audio in short chunks and emits words as it goes. The original model requires 71 MB of storage, but the project reduces its size to 26.65 MB using 2-bit and 4-bit quantisation. This allows the model weights to fit into the board’s 32 MB of PSRAM. Loading the weights from an external memory card, such as a microSD card, takes approximately 2.9 seconds at start-up.
The recognition system also supports phrase biasing, which allows users to add words and phrases in plain-text files stored on the microSD card.
To reproduce this build, users need an Espressif ESP32-P4X-Function-EV-Board (ESP32-P4X rev. v3, with 32 MB of PSRAM), a microSD card, a USB-C cable and a computer with Google Chrome or Microsoft Edge. The official repository provides a Python script to download and copy the model weights to the card. Users can then flash the firmware through a web browser or use ESP-IDF v6.1 from the command line.
Once the board is powered on for the first time, it creates the required phrase-list files on the card. Users can then speak to the board and watch the captions appear. They can also edit the phrase-list files on the card to customise the voice-control demonstrations.
According to the project, it reports a word error rate of 4.34 per cent on the LibriSpeech test-clean set and 10.26 per cent on the more challenging test-other set. The original, uncompressed Zipformer 2 model scores 3.94 per cent and 9.17 per cent, respectively, on a PC.
In terms of speed, the project takes 0.90 seconds to process 1.28 seconds of audio, with both cores running at 400 MHz. Words are finalised approximately two seconds after being spoken, while voice-control actions are triggered around 1.84 to 1.97 seconds after the command ends.
Phrase biasing also improves command recognition. It reduces the error rate across 140 test clips from 19.5 per cent to 3.4 per cent. It also achieves command-recognition rates of 94 per cent to 99 per cent when the corresponding phrase lists are used, with no false activations across 88 near-miss clips per demonstration.
The engine, firmware and supporting tools are available in the esp32-pie-asr GitHub repository under the Apache 2.0 licence. The model weights use the CC BY-SA 4.0 licence, consistent with Kroko’s licensing. A separate firmware option, called live_transcriber, provides live captions without the voice-control demonstrations. Developers can modify the firmware to adapt the system to their own applications, although changing the voice-control behaviour requires some familiarity with C and ESP-IDF.
Currently, the implementation supports English, with Arabic listed as planned work. It also requires the specified ESP32-P4X board with 32MB PSRAM, meaning lower-memory boards such as the ESP32-S3 are not suitable for this build. Its response delay makes it more appropriate for captions and predefined voice commands than natural, conversational interaction.
For Indian makers, the project could provide a starting point for offline voice-controlled switches, kiosks and classroom captioning systems, particularly in areas with unreliable internet access. However, the model has not been established as reliable for Indian English accents and does not support Hindi or other Indian languages. Phrase biasing can help with Indian names, places and technical terms, but it does not add support for a new language. Adapting the system to regional languages would require further model development and validation, alongside checks on dataset and model licence compatibility.
By bringing speech recognition to a microcontroller without relying on a separate AI accelerator or cloud processing, esp32-pie-asr offers developers and students a practical way to explore on-device AI, embedded speech processing and privacy-focused voice interfaces.
For more information, click here.
