Reddit will block access to the Internet Archive

Reddit has confirmed it will block the Internet Archive 's Wayback Machine from indexing the majority of content published on the platform. The San Francisco-based company wants to block indirect scraping by AI providers. This is the latest installment in a battle that began over a year ago .

Reddit out of the Wayback Machine

At the end of June 2024, Reddit announced that it would block all AI bots , except those owned by companies with which it has signed a licensing agreement, such as Google and OpenAI . The Californian company thus wants to prevent the scraping of public content that is used to train AI models. About a month ago, it filed a lawsuit against Anthropic .

The Internet Archive's Wayback Machine is a digital archive that preserves web pages published since May 1995. Users can therefore find older content (removed or modified) by going back in time. To create the archive, it uses a web crawler similar to Google's.

It obviously indexes Reddit content, so AI companies can leverage an alternative source if their bots are blocked by Reddit. A spokesperson said:

The Internet Archive provides a service to the open web, but we have been made aware of instances where AI companies are violating platform policies, including ours, and extracting data from the Wayback Machine.

Reddit has therefore decided to block the Wayback Machine from accessing posts, comments, and profile pages . It will only be able to index the home page, so the Internet Archive will only be able to archive the day's news headlines and most popular posts.

Reddit has informed the Internet Archive, specifying that the block will remain in place until they can secure their site. Mark Graham, director of the Wayback Machine, said they are in ongoing discussions with Reddit about the matter.