On-device AI is getting real: 27B model fits on a phone

The Bonsai 27B thread surfaced something that would have seemed implausible two years ago: a 27-billion-parameter model that runs on a phone. The discussion got into the actual mechanics, specifically that '1-bit' models are actually 1.58-bit with three values (+1, 0, -1), and that quantization tradeoffs matter a lot at this scale. The Unsloth Q2 variant was flagged as having a 5% drop in tool-call accuracy that is more significant than it sounds in practice.

This is a meaningful threshold. A 27B model on-device means local inference for tasks that previously required a server round-trip, which changes latency, cost, and privacy calculus for mobile app builders simultaneously. The discussion also mentioned an unpublished app that apparently ships this, which suggests someone is already betting on it in production.

The counterpoint in the thread was real: quantization at this level introduces subtle accuracy degradation that is easy to miss in benchmarks but painful in production, especially for tool-calling and structured output tasks. The gap between 'runs on a phone' and 'works reliably on a phone' is still being measured.