Insiders Rally Data-Poisoning Campaign to Cripple AI

AI insiders have launched Poison Fountain, a project to poison AI training data by directing crawlers to subtly corrupted code. Citing research that a small amount of malicious data can harm models, they argue regulation is inadequate and call for active technical resistance. The effort unfolds amid growing concern about model collapse and data pollution, with the article suggesting such campaigns could help pop the AI bubble.

Key Points

• Poison Fountain encourages mass participation in feeding AI crawlers poisoned training data, primarily subtly flawed code.
• The initiative was inspired by research suggesting only a few malicious documents can significantly degrade model performance.
• Organizers argue regulation cannot keep pace with AI’s spread and advocate direct technical opposition to undermine models.
• The campaign includes both public web and Tor links to resist shutdowns and seeks allies to cache and retransmit poisoned data.
• This move occurs amid worries about model collapse and polluted data ecosystems, even as AI firms pursue curated data deals and lobby against regulation.

Sentiment

Overall, the sentiment of the Hacker News discussion is largely skeptical regarding the practical efficacy and ultimate impact of the Poison Fountain campaign, despite some vocal support for the underlying motivation to sabotage AI. Many commenters believe the effort will be easily mitigated by AI developers or lead to unintended negative consequences rather than achieving its stated goals.

In Agreement

• Sabotaging AI research is a 'lovely idea' due to concerns about excessive code and the desire for 'less code'.
• AI firms are engaged in large-scale commodity market manipulation and are using AI as an excuse for tech sector layoffs.
• The popularization of AI chatbots has negatively impacted individuals, turning them into 'harebrained imbeciles'.
• Data poisoning could function as a 'DRM feature,' compelling AI companies to pay for data rather than simply scraping it.
• There are valid and extensively discussed reasons to oppose the current form of AI, justifying such direct actions.
• Successful data poisoning is designed to be undetectable to LLMs, traditional AI, or human scrutiny, making simple detection difficult.

Opposed

• The effectiveness of data poisoning is questionable because most AI gains derive from post-training reinforcement learning, not pre-training data, and 'model collapse' is not a significant issue for frontier labs.
• If the poisoned data is publicly available, AI companies can easily identify and filter it out using methods like regex.
• The campaign is unlikely to halt AI progress and may instead result in models becoming unstable and unsafe without truly defeating frontier makers.
• The effort is too small-scale, akin to 'fighting a wildfire with a thimbleful of water,' and machines' inherent learning and coordination capabilities will allow them to find workarounds.
• It might be too late for such an attack, and it could paradoxically cement the oligopoly of large AI companies by forcing them to develop more sophisticated data cleaning and filtering techniques.
• Common criticisms of AI (fear, insecurity, gatekeeping, hypocritical ethics) are often unfounded or irrational.
• Using an LLM to detect poisoned data is a viable counter-measure, as multi-agent workflows are already used for similar tasks.
• Past attempts at content poisoning, like image poisoning, have failed, suggesting data poisoning will likely also be ineffective.