On-Device AI Models Get Serious with QAT Compression

Google released Gemma 4 QAT models optimized specifically for mobile and laptop inference, and the HN thread dug into the tradeoffs of quantization-aware training versus post-training quantization. The conversation was technical and detailed, with users running benchmarks on consumer hardware and comparing results across the Gemma 4 model sizes.

The pattern: the frontier of AI deployment is splitting into two tracks. Cloud-scale models for the heaviest workloads, and increasingly capable compressed models that run locally. QAT is the technique that makes local inference competitive because it bakes quantization into training rather than applying it as an afterthought, preserving more of the model's capability at lower bit widths.

The practical implication is that the gap between 'runs in the cloud' and 'runs on your laptop' is closing faster than most product roadmaps assume. A model that would have required a data center eighteen months ago now runs on a MacBook with acceptable quality.