Anna’s Archive: Official Data Access and Donation Guide for LLMs
878
Anna’s Archive provides official bulk access methods for LLMs and requests donations from AI entities to support the preservation of the knowledge they use for training.
Web Scraping
Automated web crawling and scraping, including AI-driven data collection bots and their impact on websites and infrastructure.
Reading List
The Rise of Data Poisoning: Sabotaging the AI Slop Machines
387
Internet users are increasingly using 'data poisoning' and misinformation to sabotage AI training sets in protest of unethical web scraping.
The Bot Crisis: Why Internet Traffic is 70% Automated
236
Bot traffic is likely much higher than reported, but it can be effectively neutralized using JavaScript-based Proof of Work defenses.
GitHub - adam-s/intercept: Turn any website into a typed JSON API using self improving agents · GitHub
A framework for Claude Code that uses self-improving AI agents to transform websites into structured APIs and functional web applications.
Lightfeed Extractor: LLM-Powered Web Scraping Library
A TypeScript library for robust, LLM-powered web data extraction and browser automation.
Cloudflare Simplifies AI Data Ingestion with New Site-Wide Crawling API
487
Cloudflare's new API endpoint simplifies website-wide data extraction by automating discovery and rendering into AI-friendly formats.
Shutting Down My Self-Hosted Git After AI Scraper Overload
298
AI scrapers killed my self-hosted git, so I’ve moved everything to GitLab/GitHub and hardened my static blog’s logging.
Bots Overwhelmed Bear’s Reverse Proxy—What Broke and How It’s Now Hardened
209
Aggressive scrapers overwhelmed Bear’s reverse proxy, prompting a hardening of monitoring, capacity, and bot controls in an ongoing battle with hostile bot traffic.
Ravenous AI Crawlers Are Breaking the Web—and Driving It Behind Paywalls
213
AI crawlers’ ravenous, non-reciprocal scraping is breaking websites and pushing the open web toward paywalled fragmentation.