Humans Fail at AI Agent Oversight at Scale
A founder shared results from a game they built to test human oversight of AI agent commands. After 40,000 plays and 409,000 decisions, humans missed 1 in 3 genuine threats. That is not a marginal failure rate. For a system where the entire safety model is 'ask the user and hope they say no to the bad stuff,' this is a foundational problem.
The HN comments connected it directly to real-world agent deployments: 'It's kinda funny there is still software coming out whose security model is constantly ask the user for permission, and hope they never make a mistake. It's been tried so many times.' The analogy to browser permission dialogs and UAC prompts is apt. Users normalize them and click through. The pattern repeats at every new layer of abstraction.
The practical suggestion that emerged was custom harness and sandbox engineering rather than relying on human approval loops. That is the right call, but it shifts the burden to the builder to define safe action boundaries upfront, which is hard and often underspecified.
So what?
If you are building AI agents that take real-world actions, a human-in-the-loop approval model is not a safety net you can rely on. Founders need to design permission systems around what an agent should never be able to do regardless of user approval, not around asking users to catch bad commands. Sandboxing and capability restriction at the infrastructure level is the only durable answer.