On-device AI inference is almost good enough to ship
The benchmarking thread on pocket-scale inference and the Ask HN on replacing Claude/GPT with local models are circling the same question: when does running a model on your own hardware stop being a hobbyist experiment and become a legitimate production choice? The pocket inference thread notes Apple has been stingy with RAM, and that RAM prices are expected to climb into 2030, which makes the next generation of on-device models harder to ship than the hardware roadmaps suggest. The local model thread has people reporting real results with Qwen 3.6 27b dense at roughly Claude Haiku quality, and one person running DeepSeek V4 Flash on dual RTX Pro 6000 Blackwell cards at 160 tokens per second.
The pattern here: the gap between frontier cloud models and the best local options has narrowed enough that the conversation has shifted from 'can I?' to 'should I for this specific use case?' Coding assistance, in particular, keeps coming up as the test case, because the feedback loop is tight and the privacy argument is real.
The counterpoint is that enterprise tooling for local models is still thin. Several commenters note there's no good way to evaluate which model fits a given task without just running experiments yourself. OpenRouter is being used as a middle path, letting people swap models without committing to local infrastructure.
So what?
If you're building a product that uses LLM inference, the cost and privacy math for local or self-hosted models is now worth running seriously, not just for security-sensitive use cases. The catch is that tooling for model selection and evaluation is immature, so budget time for experimentation before you commit to an architecture.