# Publishers Blocking Web Archives: A Threat to Digital Heritage in the AI Era
Large media outlets, including The New York Times and The Guardian, are putting up technical barriers to stop the Internet Archive from indexing their sites. These go beyond the standard robots.txt file and are aimed at preventing web scraping for AI model training. But this approach risks losing historical snapshots of web pages—the only remaining source of original content.
EFF senior analyst Joe Mullin compares this to banning libraries from keeping copies of newspapers. Since the 1990s, the Internet Archive has been capturing snapshots of news sites, including edits and deleted articles. These blocks cut off historians, journalists, and researchers from authentic versions of publications.
Role of the Internet Archive in Preserving Web History
The Internet Archive operates as a non-profit digital library, building indexes and copies of web resources. This is much like how search engines work, where copying content is recognized as fair use under US case law.
- The only source of originals: Many articles get edited or deleted without notice; the archive preserves their initial versions.
- Long-term storage: Nearly 30 years of data used for academic research and fact-checking.
- Non-commercial status: The organization doesn't engage in commercial AI training, focusing instead on preserving cultural heritage.
Publishers are justifying these blocks with lawsuits against AI companies over unauthorized scraping. The New York Times has already filed several cases, demanding compensation for content used in model training.
Why Blocking Is the Wrong Approach
Mullin stresses that these measures won't stop AI bots but will wipe out decades of documentation. Search engines are legally protected because indexing requires copying content. Archives should similarly retain the right to cache pages for the public good.
Court rulings on AI scraping might impose limits, but they shouldn't apply to non-profit libraries. Shutting out archives erases the historical record, sacrificing data in a fight that archivists aren't even part of.
Legal Aspects of Indexing and Archiving
In the US, fair use covers creating search indexes and archives:
- Purpose and character: Non-commercial and educational—strongly favors fair use.
- Nature of the work: Factual media content is well-suited for archiving.
- Amount: Only the copies necessary for indexing.
- Market effect: Archives don't compete with originals; they preserve them.
EFF argues that precedents for Google and Bing apply to the Internet Archive. Restrictions on AI shouldn't touch these established practices.
Key Takeaways
- Blocking archives doesn't stop AI scraping but destroys historical records of web content.
- The Internet Archive is a vital tool for verifying changes in media publications.
- Fair use protects non-commercial archiving, just like search engines.
- Publishers risk permanently losing digital heritage in their battle with AI companies.
- EFF recommendation: Differentiate access for commercial AI versus non-profit libraries.
Implications for Digital Heritage
Sacrificing public data to control AI is an irreversible mistake, says Mullin. Losing archival copies will make it harder to analyze propaganda, fake news, and how narratives evolve. AI developers and researchers studying historical data will face a shortage of authentic sources.
For IT pros, this is a wake-up call to rethink web scraping and archiving practices. Expect new protocols like robots.txt rules that differentiate bots by User-Agent, or decentralized archives on blockchain for better resilience.
— Editorial Team
No comments yet.