AI Poisoning Attacks on Search and Chatbots Are Now Widespread
A thread on hackers influencing ChatGPT, Gemini, and Google AI Overview to direct users to scam centers landed alongside continued discussion of AI-generated content flooding the web. The mechanism is straightforward: create websites with false information, wait for AI crawlers to index it, and the misinformation surfaces in AI-generated answers to user queries. Commenters noted this is not just hackers, it is a structural problem: because legitimate sites block AI crawlers, the crawlers end up disproportionately ingesting content from sites that do allow access, which skews toward low-quality and adversarial content.
This connects directly to the arXiv funding thread, where the editor noted they are struggling to keep up with an onslaught of AI-generated papers. The signal-to-noise problem is not confined to one platform or one type of content. It is a systemic issue across the information ecosystem that AI both accelerates and is victimized by.
The key insight is that AI systems trained on or grounded in live web data are now actively adversarial attack surfaces. The attack requires no technical sophistication: publish content, wait.
So what?
If your product uses retrieval-augmented generation or any grounding against live web data, you need a threat model that includes adversarial content injection. This is not theoretical. For founders building AI products that answer factual questions about companies, prices, or services, you are one well-executed poisoning campaign away from a trust crisis.