LLM Crawlers Are Breaking the Open Web

The Internet Archive posted an update saying Wayback Machine has been hit by waves of high-volume automated traffic and they have put protections in place to keep the service running. They did not name the culprits, but the comment thread is not subtle: AI training crawlers are widely assumed to be the cause. One commenter suggested the Archive should offer a paid API endpoint for crawlers rather than playing whack-a-mole with blocks.

This is not an isolated complaint. The Cloudflare-Spain situation, where a football rights block on Cloudflare's network accidentally blocked Docker Hub, and the general discussion around Cloudflare increasingly blocking non-mainstream browsers, all point to the same structural problem: a handful of infrastructure chokepoints are being asked to make real-time policy decisions about who counts as a legitimate user at massive scale, and they are getting it wrong constantly.

The Internet Archive is a public-good institution running on donations. It is a particularly brutal target because it cannot easily monetize the traffic the way a commercial CDN might. The community mood is frustrated but practical, with several people noting that no technical block will stop determined crawlers for long.