LLM Inference Speed Race Heating Up on Commodity Hardware

A thread on real-time LLM inference hitting 3,000 tokens per second on standard GPUs generated genuine interest on HN, though with healthy skepticism. The main pushback: the benchmark is against a 2B model, and comparisons to Groq are unfair because Groq runs much larger models. Still, the directional signal is real. People are actively working on making fast inference available without specialized hardware.

This sits alongside the Claude Code configuration thread, which revealed a lot of undocumented knobs for tuning AI agent behavior, and the AI code review thread where teams are running orchestrated multi-model review pipelines locally. The theme is builders moving from 'use the cloud API' to 'run this thing myself, configure it precisely, and own the infrastructure.'

The Groq comparison debate is actually the interesting part. Groq's business model depends on inference speed being a moat. If commodity GPUs close the gap even on smaller models, the pricing pressure on inference providers will be significant.