Tiny models running on microcontrollers are becoming real

Two threads this week push the same frontier: a 28.9M parameter LLM running on an $8 ESP32 microcontroller at 9.7 tokens per second, and a 9.36M parameter TTS model from Inflect that produces complete voice synthesis under 10M parameters. Both are hitting a threshold where useful language and audio inference can happen on hardware that costs less than a cup of coffee, with no network connection required.

The pattern: the edge inference story is no longer theoretical. The ESP32 project uses a per-layer embedding trick to fit a functional LLM on a chip with kilobytes of RAM. Someone in the thread noted that pairing this with a similarly tiny TTS model means you could build a fully local voice assistant on an ESP32 with zero cloud dependency. The Inflect model separately impressed commenters with audio quality that seemed implausible at that parameter count.

This matters beyond hobbyist projects. Industrial IoT, medical devices, offline-capable consumer hardware, and any context where latency or data sovereignty rules out cloud inference are all immediately addressable with this class of model. The missing piece is still STT, but even that is closing fast.