Running a dense, multi-million-parameter transformer on an $8 microcontroller sounds like an exercise in immediate out-of-memory panics. The classic hardware ceiling for edge devices like the ESP32-S3 has long hovered around a modest 260,000 parameters. Beyond that, the 512KB internal SRAM instantly chokes. Developer Slava S. fundamentally shattered that limit by deploying a 28.9-million-parameter model on bare-metal silicon, running locally at a sustained 9.5 tokens per second.
That is a clean 100x scale jump over previous attempts. This was achieved without cloud offloading, tethered co-processors, or esoteric hardware hacks. The solution relies on a surgical rethink of memory tiering inspired by Google Gemma, matching transformer layer execution directly to physical silicon buses.
The Hardware Bottleneck: The SRAM Trap
To understand why this execution is notable, look at the memory topology of the standard ESP32-S3.
The chip gives you 512KB of ultra-fast internal SRAM, paired externally with 8MB of Octal PSRAM and 16MB of NOR Flash. Standard inference engines attempt to keep the full operational model in fast addressable memory. If the model does not fit in fast RAM, inference usually collapses due to bus latency or fails to initialize entirely.
Slava S. recognized a crucial asymmetric property of language models: compute density is not uniform across parameter types.
While dense transformer layers require high-bandwidth read-write operations during self-attention, the massive embedding matrix is primarily a sparse lookup table. It is read-heavy, not compute-heavy. Treating an embedding weight the same way as an attention weight wastes your most expensive memory budget.
Architectural Breakthrough: Per-Layer Embeddings
To shatter the 260k-parameter ceiling, the project ported the Per-Layer Embeddings concept introduced in Google Gemma architectures (specifically Gemma 3n and Gemma 4) down to bare-metal embedded C.
[ FLASH (16MB) ] ------------> Fetches ~6 rows/token (~450 B)
25M Parameter Lookup Table
|
v
[ PSRAM (8MB) ] -------------> KV Cache, Activations, Output Head
|
v
[ SRAM (512KB) ] ------------> Active Transformer Compute Core
Instead of forcing the entire network into memory, the architecture bifurcates the parameter distribution across the hardware's three storage tiers:
- Flash Storage (16MB): Houses the massive 25-million-row parameter embedding table. Because the table is indexed sparsely per generation step, the engine only fetches the exact required rows: approximately 6 rows, or roughly 450 bytes per token.
- PSRAM (8MB): Holds the dynamic runtime footprint, including intermediate activations, KV cache state, and the final output head.
- Internal SRAM (512KB): Retains the execution-critical reasoning core (the actual transformer blocks). Because this compute-heavy loop stays pinned inside internal SRAM, the core mathematical operations proceed without stalling on slower memory buses.
The total footprint comes out to 14.9MB using 4-bit quantization. By streaming tiny slices of the embedding table on demand from raw flash while locking active attention blocks into SRAM, the bulk of the model never needs to be resident in RAM simultaneously.
On-Chip Telemetry and Specifications
The engineering balance pays off in measurable throughput. Streaming tokens directly to a connected hardware display yields consistent interactive performance without relying on an external server.
| Metric / Component | Specification | Notes & Role |
|---|---|---|
| Model Size | 28.9M Parameters | 25M isolated inside Flash lookup tables |
| Quantization Format | 4-bit Quantized | Total image size: 14.9MB |
| Silicon Target | ESP32-S3 | Estimated module cost: ~$8 |
| On-Chip Memory | 512KB SRAM / 8MB PSRAM / 16MB Flash | Tiered hierarchy allocation |
| End-to-End Speed | ~9.5 tokens/sec | Output directly driving display |
| Pure Compute Speed | ~9.7 tokens/sec | Raw internal inference step latency |
| Network Interface | Zero / Air-gapped | 100% autonomous local execution |
| Core Architecture | Per-Layer Embeddings | Adapted from Google Gemma |
Capabilities vs. Architectural Reality
Engineers looking to build production systems must separate mathematical architecture from operational capability.
The underlying model was trained exclusively on the TinyStories dataset (Microsoft Research). Consequently, the device excels at generating synthetically structured, coherent, simple narratives.
[Input Prompt] -> (ESP32-S3 Inference) -> "The little girl saw a bird..."
It is vital to be precise about what this is not:
- It is not an instruction-following assistant.
- It will not answer arbitrary trivia, write code, or process logical queries.
- It cannot function as an embedded knowledge retrieval database.
The underlying reasoning core remains compact. The memory tiering bypasses storage constraints, but it does not alter the fundamental cognitive limits imposed by the parameter capacity of the active transformer layers. The achievement here is purely systems-level and architectural, proving that deep memory tiering can push multi-million-parameter networks onto sub-ten-dollar edge microcontrollers.
Open-Source Lineage & How to Run It
The project builds upon two core foundations: Andrej Karpathy's llama2.c, which popularized stripped-down C execution loops for small models, and the TinyStories dataset (Eldan & Li, arXiv:2305.07759).
The repository is published under the permissive MIT license. The implementation includes full hardware-level documentation and an honest version-control record (including early parameter-counting corrections documented transparently in the repo's measurement notes).
You can inspect the implementation across these project paths:
firmware/esp32_llm/README.md: Step-by-step flashing guides, pinout maps, and wiring specifications.src/andexperiments/: Pure training loops, ablation testing suites, and quantization scripts.RESULTS.md: Empirical profiling traces, parameter breakdowns, and on-chip hardware benchmarks.
By treating memory hierarchy as an explicit design primitive rather than a passive run-time target, this project outlines a clear roadmap for running specialized neural architectures right at the edge of embedded hardware.