Running big models locally is getting real
Several hardware and software threads point the same way. Intel's D270 accelerator (Crescent Island) was announced with up to 480 GB of LPDDR5X, and commenters immediately asked about price against a Mac and whether you can even buy one. Janas runs LLMs larger than RAM by streaming experts from SSD, with a demo at 62 tokens per second, and a commenter notes there are now many projects doing this and someone should merge the best ideas. A transcription demo does an hour of audio in about 13 seconds on a MacBook Air.
A related Ask HN on replacing Claude or GPT with a local model for daily coding had real answers. One user said Qwen 3.6 27b dense feels about like Claude Haiku 4.5, maybe Sonnet on some tasks. Others said tokens per second on an M4 with Gemma 4 is still far below the cloud, and enterprise tooling for picking and running models is missing.
The through-line: the bottleneck has moved from compute to memory capacity and bandwidth. People are routing around it with SSD streaming and giant LPDDR pools rather than waiting for bigger GPUs.
So what?
Local inference is now good enough for mid-tier coding and transcription tasks. If your margin depends on resale of frontier API calls, plan for customers who run cheaper models on their own hardware. If you build tooling, the gap is model selection, setup and packaging, not raw speed.
Read these
Intel D270 GPU AI Accelerator–Up to 480 GB LPDDR5X
Janas: Run LLMs larger than RAM by streaming experts from SSD
An hour of audio transcribed in about 13s on MacBook Air
Ask HN: Has anyone replaced Claude/GPT with a local model for daily coding?