Cloudflare will ask Google to separate bots

Cloudflare recently announced the automatic default blocking of AI bots for new domains and a Pay-per-Crawl option (paying for access to content). Several SEO experts and website owners have expressed doubts about the effectiveness of these solutions. Matthew Prince , CEO of the California-based company, provided an explanation on X.

Google must separate the AI ​​bot from the standard one

It's well known that generative AI models are trained on content published online. In most cases, publishers are not paid anything . Due to indiscriminate "scraping," users read content through various chatbots, thus avoiding visiting the source, and publishers earn less from advertising.

Cloudflare recently activated an automatic AI bot block for new domains. However, there's a small problem. Several AI-enabled search engines use the same bot for indexing sites and collecting content to train their AI models. Google, for example, uses two main bots: Googlebot for indexing and Google-Extended for Gemini.

Matthew Prince, Cloudflare CEO, confirmed that Google Extended is blocked by default. However, the Mountain View company has integrated AI Overview and AI Mode into the search engine. To prevent these two features from accessing content, Googlebot should be blocked. This will obviously cause the site to be de-indexed (essentially, it's as if it no longer exists).

Matthew Prince hopes Google will offer a way to block AI Overview and AI Mode without blocking indexing. Without an amicable agreement, Cloudflare could seek legislation requiring Google to separate the services, but this seems an impractical solution.

Cloudflare's automatic bot blocking could have another side effect. There's a risk that bots used for academic research and security scans could be blocked. The work of The Internet Archive could also be hampered, preventing it from preserving web pages via the popular Wayback Machine.