LLM Inference Optimization Techniques Have Stalled

The efficient frontier of LLM inference thread (49529898) surfaces a striking observation: the core techniques for optimizing inference, speculative decoding, quantization, batching strategies, have barely changed in years. Precision has crept down slightly, speculative decoding acceptance rates improved modestly, but the fundamental toolkit is the same. The post argues there's a 2D frontier of throughput vs. latency, and commenters add that quality is a third dimension that breaks the clean optimization picture.

This matters for builders because it means squeezing more performance out of inference is increasingly about deployment architecture and hardware choices rather than algorithmic breakthroughs. The gains are incremental. That's good news for incumbents with optimized infrastructure and bad news for anyone hoping a new technique will make their cost structure suddenly competitive.

The local model discussion (49529132) reinforces this. People running models on M4 Mac Minis are struggling to share performance numbers because results vary so much by workload. The efficiency gains are real but hard to generalize.