An open-source ESP32-S3 project runs an offline homework assistant that listens, answers and speaks without cloud AI.
A single ESP32-S3 board can listen to a question, process it using an on-device language model and speak an answer without sending voice recordings to a cloud server. Called Ivy AI, the open-source project combines speech recognition, a wake-word detector, a compact language model and text-to-speech on a microcontroller board with a display, microphone and speaker. Its developer designed it as an offline voice assistant for simple questions, including those related to children’s learning.
The project runs on the Freenove FNK0104B, which combines an ESP32-S3R8 microcontroller with 8MB of PSRAM and 16MB of flash storage. It also integrates a 2.4-inch, 240×320-pixel LCD, a touch controller, an ES8311 audio codec, a microphone, a speaker, a microSD card slot and a lithium-ion battery charging circuit. The board provides the hardware needed for a standalone voice assistant without requiring a separate Raspberry Pi or external computer to process questions.
Ivy’s processing pipeline begins with Espressif’s WakeNet9 wake-word detector, which waits for the phrase “Hey Ivy”. A Conformer-based connectionist temporal classification (CTC) speech-recognition engine converts the spoken question into text. A small language model then generates a response, and SVOX Pico converts that response into spoken audio. The project uses Espressif’s ESP-IDF development framework and is written in C rather than relying on the Arduino framework.
The language model is the most demanding part of the system. The project’s default TinyTalk 2 8M v8 model is listed at approximately 19.7 million parameters and occupies around 6.5MB in its quantised form. The speech-recognition model requires approximately 5.4MB of memory, meaning the two cannot simply remain loaded in the board’s 8MB of PSRAM at the same time. Ivy addresses this limitation by allocating memory for the processing stages and loading the required model when each stage begins. Speech recognition and language generation therefore take turns using the available memory.
The developer reports that the default language model takes about 694 milliseconds to load and generates text at approximately 5.4–5.9 tokens per second. A smaller 6.4-million-parameter model reaches around 7.3 tokens per second, while a much smaller model with approximately 260,000 parameters can reach 30 tokens per second. These figures demonstrate how model size affects performance on a memory-constrained microcontroller, although generation speed alone does not indicate the quality or correctness of an answer.
Speech recognition remains a bottleneck. According to the project’s measurements, transcribing 1.4–2.9 seconds of speech takes approximately 2.7–4.6 seconds. The developer estimates that a complete spoken interaction takes around 14–17 seconds, including listening, transcription, model loading, answer generation and speech synthesis. The wake-word detector and voice-activity detection also have limitations: the latter can occasionally clip the final syllable of a question.
Ivy can also run compatible small GGUF language models stored on the microSD card. The repository documents a model-loading process and provides tools for transferring model files to the board. However, compatibility depends on the model’s architecture, vocabulary and memory requirements; users cannot assume that every small language model will work simply because it is available in GGUF format.
The project is licensed under GPL-3.0-or-later, but that licence does not automatically cover every bundled component or model. Its speech-recognition engine, speech-recognition weights, text-to-speech engine, language-model engine and model weights have separate licensing conditions. The developer specifically warns that some language-model weights inherit non-commercial restrictions from their training data. Anyone adapting Ivy for a commercial product must therefore review the individual licences rather than relying solely on the repository’s main licence.
The system is not a replacement for a general-purpose AI assistant. Its small language model has limited capabilities, and the developer’s evaluation shows a substantial difference between questions phrased as they appeared during training and alternative phrasings of the same questions. Its speech-recognition speed, English-language support and occasional transcription errors also limit its suitability for unrestricted conversations. It is better understood as an experimental embedded-AI project than as a finished educational product.
For Indian makers, Ivy offers a starting point for developing offline voice interfaces that do not depend on cloud-based inference. However, the current implementation uses US English and a curriculum that includes US-specific geography. Adapting it for Indian school syllabuses, Indian English or regional languages would require changes to the language model, speech-recognition system and voice output, followed by testing on the target hardware.
For more information, click here.
