Running a dense, multi-million-parameter transformer on an $8 microcontroller sounds like an exercise in immediate out-of-memory panics. The classic hardware ceiling for edge devices like the ESP32-S3 has long hovered around a modest 260,000 parameters. Beyond that, the 512KB internal SRAM instantly chokes. Developer Slava S. fundamentally shattered that limit by deploying a **28.9-million-parameter model** on bare-metal silicon, running locally at a sustained **9.5 tokens per second**.

That is a clean 100x scale jump over previous attempts. This was achieved without cloud offloading, tethered co-processors, or esoteric hardware hacks. The solution relies on a surgical rethink of memory tiering inspired by Google Gemma, matching transformer layer execution directly to physical silicon buses.

---

## The Hardware Bottleneck: The SRAM Trap

To understand why this execution is notable, look at the memory topology of the standard ESP32-S3. 

The chip gives you 512KB of ultra-fast internal SRAM, paired externally with 8MB of Octal PSRAM and 16MB of NOR Flash. Standard inference engines attempt to keep the full operational model in fast addressable memory. If the model does not fit in fast RAM, inference usually collapses due to bus latency or fails to initialize entirely. 

Slava S. recognized a crucial asymmetric property of language models: compute density is not uniform across parameter types. 

While dense transformer layers require high-bandwidth read-write operations during self-attention, the massive embedding matrix is primarily a sparse lookup table. It is read-heavy, not compute-heavy. Treating an embedding weight the same way as an attention weight wastes your most expensive memory budget.

---

## Architectural Breakthrough: Per-Layer Embeddings

To shatter the 260k-parameter ceiling, the project ported the **Per-Layer Embeddings** concept introduced in Google Gemma architectures (specifically Gemma 3n and Gemma 4) down to bare-metal embedded C. 

```
[ FLASH (16MB) ] ------------> Fetches ~6 rows/token (~450 B)
  25M Parameter Lookup Table
           |
           v
[ PSRAM (8MB) ] -------------> KV Cache, Activations, Output Head
           |
           v
[ SRAM (512KB) ] ------------> Active Transformer Compute Core
```

Instead of forcing the entire network into memory, the architecture bifurcates the parameter distribution across the hardware's three storage tiers:

1. **Flash Storage (16MB):** Houses the massive 25-million-row parameter embedding table. Because the table is indexed sparsely per generation step, the engine only fetches the exact required rows: approximately 6 rows, or roughly 450 bytes per token.
2. **PSRAM (8MB):** Holds the dynamic runtime footprint, including intermediate activations, KV cache state, and the final output head.
3. **Internal SRAM (512KB):** Retains the execution-critical reasoning core (the actual transformer blocks). Because this compute-heavy loop stays pinned inside internal SRAM, the core mathematical operations proceed without stalling on slower memory buses.

The total footprint comes out to **14.9MB using 4-bit quantization**. By streaming tiny slices of the embedding table on demand from raw flash while locking active attention blocks into SRAM, the bulk of the model never needs to be resident in RAM simultaneously.

---

## On-Chip Telemetry and Specifications

The engineering balance pays off in measurable throughput. Streaming tokens directly to a connected hardware display yields consistent interactive performance without relying on an external server.

| Metric / Component | Specification | Notes & Role |
| :--- | :--- | :--- |
| **Model Size** | 28.9M Parameters | 25M isolated inside Flash lookup tables |
| **Quantization Format** | 4-bit Quantized | Total image size: 14.9MB |
| **Silicon Target** | ESP32-S3 | Estimated module cost: ~$8 |
| **On-Chip Memory** | 512KB SRAM / 8MB PSRAM / 16MB Flash | Tiered hierarchy allocation |
| **End-to-End Speed** | ~9.5 tokens/sec | Output directly driving display |
| **Pure Compute Speed** | ~9.7 tokens/sec | Raw internal inference step latency |
| **Network Interface** | Zero / Air-gapped | 100% autonomous local execution |
| **Core Architecture** | Per-Layer Embeddings | Adapted from Google Gemma |

---

## Capabilities vs. Architectural Reality

Engineers looking to build production systems must separate mathematical architecture from operational capability. 

The underlying model was trained exclusively on the **TinyStories** dataset (Microsoft Research). Consequently, the device excels at generating synthetically structured, coherent, simple narratives. 

```
[Input Prompt] -> (ESP32-S3 Inference) -> "The little girl saw a bird..."
```

It is vital to be precise about what this is not:
* It is **not** an instruction-following assistant.
* It will **not** answer arbitrary trivia, write code, or process logical queries.
* It cannot function as an embedded knowledge retrieval database.

The underlying reasoning core remains compact. The memory tiering bypasses storage constraints, but it does not alter the fundamental cognitive limits imposed by the parameter capacity of the active transformer layers. The achievement here is purely **systems-level and architectural**, proving that deep memory tiering can push multi-million-parameter networks onto sub-ten-dollar edge microcontrollers.

---

## Open-Source Lineage & How to Run It

The project builds upon two core foundations: Andrej Karpathy's `llama2.c`, which popularized stripped-down C execution loops for small models, and the TinyStories dataset (Eldan & Li, arXiv:2305.07759).

The repository is published under the permissive MIT license. The implementation includes full hardware-level documentation and an honest version-control record (including early parameter-counting corrections documented transparently in the repo's measurement notes).

You can inspect the implementation across these project paths:
* `firmware/esp32_llm/README.md`: Step-by-step flashing guides, pinout maps, and wiring specifications.
* `src/` and `experiments/`: Pure training loops, ablation testing suites, and quantization scripts.
* `RESULTS.md`: Empirical profiling traces, parameter breakdowns, and on-chip hardware benchmarks.

By treating memory hierarchy as an explicit design primitive rather than a passive run-time target, this project outlines a clear roadmap for running specialized neural architectures right at the edge of embedded hardware.

Discover how a developer fit a 28.9M parameter LLM onto an $8 ESP32-S3 using Per-Layer Embeddings, bypassing the SRAM ceiling to hit 9.5 tokens/sec entirely on-device.

Cracking the SRAM Wall: How to Run a 28.9M Parameter LLM on an $8 ESP32-S3

Reggi

Eliminate context switching and manage custom domain email directly inside Gmail. Configure unified inbound ingestion and Send As permissions seamlessly.

The Unified Inbox Architecture: Routing Custom Domain Webmail Through Gmail

Eliminate manual rotation bottlenecks, thermal lag, and biological variance by building an automated, headless AC-powered egg incubator setup.

Stop Acting as a Manual Cron Job: Engineering an Automated AC-Powered Incubator

Ditch mains AC for a stable, low-voltage DC egg incubator. Learn how dedicated temp and humidity control modules optimize poultry yields.

Why AC Mains Are Ruining Your Hatch Rates: Building a Reliable Low-Voltage DC Incubator

A technical systems breakdown of battery-powered ride-on car assembly for ages 2 to 7 (50kg max), covering safety envelopes, parental remote control, and multimedia architecture.

Architecting the Scaled EV: Remote Override Logic and Multimedia Integration for Electric Ride-On Chassis

Learn how to build a reliable smart power strip using NodeMCU and Blynk for wireless and voice-controlled electrical management.

Why Remote Power Switching Fails in Hardware Prototypes and How NodeMCU with Blynk Solves It

Fix persistent macOS bugs, wipe disk bloat, or prep for resale. Master Internet Recovery (Command+Option+R), APFS reformatting, and clean installation.

#tinyml

Getting Lastest Post..

Trending This Month