AI benchmark credibility is collapsing under its own weight

Two threads tackled the benchmark problem from different angles. Anthropic's 'Separating signal from noise in coding evaluations' post was read by commenters as an admission that other labs had learned to overfit SWE-Bench Pro better than Anthropic had. Separately, Databricks published benchmarks of coding agents against their multi-million line internal codebase, finding that many models are now competitive at the top tier, that open source has caught up significantly (GLM 5.2 was called out specifically), and that the harness around the model matters as much as the model itself.

The pattern: public benchmarks are now so gamed that serious evaluators are building private ones. The Databricks post is a good example of what trustworthy evaluation looks like: a real codebase, real tasks, costs measured, and no model vendor involved in the design. The HN comment about not knowing how to replicate a 1990s Super Soaker design is funny but the point is real: frontier models fail on problems with no training data.

Grok 4.5 launching hours after GPT's latest release, with commenters noting the timing pattern across all labs, adds to the noise. When every lab releases on the same day with competing benchmark claims, the numbers stop meaning anything.