When Your Language Model Fits in a Microcontroller: 28.9M Parameters on an ESP32-S3
You've probably seen the demos: someone runs a tiny language model on a Raspberry Pi or a laptop, and it feels impressive. But what if you could run it on something the size of a postage stamp, with no network connection, no operating system, and less RAM than a single browser tab? That's exactly what this project does, and the trick it uses is genuinely clever. It's a 28.9 million parameter language model running entirely on an ESP32-S3 microcontroller, generating text at nearly 10 tokens per second on a small attached screen.
What It Does
The project, esp32-ai, is a complete setup for running a language model on an ESP32-S3 chip. That chip has 512KB of SRAM, 8MB of PSRAM, and 16MB of flash. For context, your phone probably has thousands of times more memory. Yet this thing generates coherent short stories at 9.88 tokens per second, with all computation happening on-device. Nothing goes to a server.
The model itself is trained on TinyStories, a dataset of simple children's stories, so it writes short narrative text rather than answering questions or writing code. There's also a second model called Barista, trained for espresso-related question answering, which is a delightfully specific use case.
The hardware setup is straightforward: the ESP32-S3 board with a small screen wired to it. The software side handles fetching models, verifying them, and deploying them to the board. It's a complete pipeline, not just a demo binary.
Why It's Cool
The interesting part isn't the model itself—28.9 million parameters is tiny by modern standards. What's interesting is how it fits at all.
The memory hierarchy trick. A microcontroller has very little fast memory. The ESP32-S3's 512KB of SRAM is where activations and norm weights live, since those get touched constantly during generation. The dense core and output head, scanned once per token position, sit in the slower PSRAM. The bulk of the model—a 25 million parameter embedding table—stays in flash memory, which is huge but incredibly slow.
The Per-Layer Embeddings idea. This comes from Google's Gemma 3n architecture. Instead of computing embeddings, the model reads them from a lookup table. Since each token only needs a few rows from that table—about 450 bytes per token—the model can sample from flash lazily, pulling only what it needs at each step. Most of the model is never loaded into RAM at all. It just sits in flash, waiting to be sampled a little at a time.
The honest framing. The project is refreshingly upfront about what this model can't do. It won't answer questions, follow instructions, or know facts. The memory trick doesn't make a small model smarter—it just lets a small model fit in a very constrained environment. The value here is architectural: demonstrating how to map a modern LLM architecture onto a microcontroller's memory layout.
The practical deployment pipeline. The scripts are well thought out. fetch_model.sh downloads and verifies models against pinned SHA-256 hashes and byte sizes, cross-checking the release's own metadata. It won't install anything unless every check passes, so a failed download leaves your existing setup untouched. deploy.sh works from whatever's already in your artifacts directory, running header generation, gates, compilation, and flashing.
How to Try It
Getting started is a two-step process, and the separation is deliberate: downloading a model and deploying it to your board are independent operations.
First, clone the repository and fetch a model:
scripts/fetch_model.sh barista # download, verify, install into artifacts/
Then deploy it to your board:
scripts/deploy.sh barista # generate headers, run gates, compile, flash
The tinystories model works the same way, just substitute the name. You can only have one model on the board at a time, since deploying replaces whatever's there.
One thing to note: deploy.sh runs two header tools through uv, which will fetch a pinned wheel on first use. The README mentions the model files are 14.9MB at 4-bit quantization, and you'll need an ESP32-S3 with at least 8MB PSRAM and 16MB flash to make this work.
The models themselves are hosted on Hugging Face: Barista for espresso QA and TinyStories for story generation.
Final Thoughts
This project is best for embedded developers who want to see what's possible at the extreme low end of the hardware spectrum, or for anyone curious about how memory hierarchies shape what models can run where. It's a proof of concept, not a production LLM—but it's a proof of concept that shows a genuinely clever architectural idea working in a constrained environment.
The takeaway isn't that you'll run ChatGPT on a microcontroller anytime soon. It's that the gap between "large language model" and "tiny embedded device" is narrower than you might think, and that clever memory management can do a lot of the heavy lifting. If you've got an ESP32-S3 lying around, this is a fun weekend project that'll give you a new appreciation for what those little chips can do.
Follow @githubprojects for more developer tools and open source projects.