Running big models locally on minimal hardware is becoming real
A Show HN post for an open-source engine running Gemma 4 26B in 2 GB of RAM on any M-series Mac is getting genuine excitement. The headline numbers: 5-6 tokens per second on an 8 GB M2 MacBook Air and 31-35 tokens per second on an M5 MacBook Pro. The engine streams model weights from SSD rather than loading them fully into RAM, and commenters were surprised that expert selection in the mixture-of-experts architecture does not thrash the SSD as badly as intuition suggests.
This sits next to Kimi K3-256k, which is a 256k context window API change, and the broader availability of capable models through APIs. But the local-first angle is distinct: running a 26B model on hardware someone already owns, with no API costs and no data leaving the device, changes the economics and privacy calculus for a whole class of applications.
The counterpoint from commenters: if you have enough RAM to load the model fully, you probably should not use the SSD-streaming engine. This is a tool for constrained environments, not a general replacement for full in-memory inference.
So what?
Privacy-sensitive applications, offline tools, and anything where API costs compound at scale now have a credible local inference path for models that would have required a server six months ago. If your product currently sends user data to an external model API and that makes customers nervous, this is worth prototyping against.