Local model quality gap closes at 4-bit quantization

A detailed benchmarking thread on Qwen3.8 27B quantizations found that model quality holds up well down to 4-bit, with meaningful degradation only appearing at 2-bit and a collapse at 1-bit. The practical implication is that you can run a strong 27B model on a sub-16GB GPU, covering the RTX 5080, 5070 Ti, 5060 Ti, and similar cards that are common in the current generation.

A separate Ask HN thread on replacing Claude and GPT with local models for daily coding showed real people doing exactly this, with mixed results depending on hardware. Apple M1 Max users found Qwen3.8 27B too slow for interactive use, while people with newer Apple silicon or Blackwell-era Nvidia cards reported acceptable speeds. The thread on Desert Ant Labs, which ships fast local inference models for Apple devices, added another data point: there's a genuine market forming around on-device AI that doesn't phone home.

The counterpoint is that local models still lag on reasoning tasks and lack the tooling that makes cloud models productive. But the gap is closing faster than most people expected a year ago.