Model benchmarks are breaking down as a trust signal

The 'Benchmarkpocalypse' thread argues that LLM benchmarks are being gamed so thoroughly that they've stopped being meaningful signals of real-world capability. The discussion connects to a broader pattern: GPT-5.6 Sol launched with strong benchmark numbers and vision claims, but HN users immediately pushed back with anecdotal evidence that it's worse at simple tasks than GPT-5.4, and that it over-complicates basic requests. One commenter asked it to write a user todo and got a four-page essay.

The pattern here: there's now a recurring cycle where a new model drops, benchmarks look good, early adopters report regression on everyday tasks, and the community spends a week figuring out what's actually true. Benchmarks measure what they measure, and the things they measure are increasingly optimized for, not representative of.

The metamorphic testing suggestion in the thread is genuinely interesting as an alternative approach, but it's niche and hard to standardize. For now, the practical reality is that founders building on top of these models need their own internal evals, not borrowed benchmark scores.