Agent benchmarks and harness design get serious

A video on RuneScape Bench, MMOs as environments for agents, was called 'a goldmine of ideas on how to build agent harnesses and benchmark them.' One commenter suggested casual resource-collection games, which are a series of mini-games, could make good model training environments. Elsewhere the front page had an explainer on why speculative decoding is fast, with a robotics commenter drawing a parallel to action chunking. It also had a 10-month write-up on building the most human-like chess bot, and Maxwell, a 'new kind of model that sits between code and hardware' that commenters said the homepage explained poorly.

The pattern: the interesting work is moving from model quality to the environment around the model, meaning the harness, the task, the speed tricks and the evaluation. Games are attractive because they have clear state, long horizons and cheap resets.

The counterpoint from the Maxwell thread is that fresh framing still needs a clear explanation. A manifesto that excited people didn't tell them what they were looking at.