Cloudflare offers to make AI pay to crawl websites

Cloudflare will block AI crawlers from accessing new customers’ websites without permission starting July 1 and is testing a way to make AI pay for the data it gathers.

Furthermore, website owners can now decide who crawls their sites, and for what purpose, and AI companies can reveal via Cloudflare whether the data they gather will be used for training, inference, or search, to help owners decide whether to allow the crawl.

The company began enabling its customers to choose to block AI crawlers in July 2024. Since then, it said, over one million customers have opted in.

“For decades, the Internet has operated on a simple exchange: search engines index content and direct users back to original websites, generating traffic and ad revenue for websites of all sizes. This cycle rewards creators that produce quality content with money and a following, while helping users discover new and interesting information,” Cloudflare said in its announcement. “That model is now broken. AI crawlers collect content like text, articles, and images to generate answers, without sending visitors to the original source — depriving content creators of revenue, and the satisfaction of knowing someone is reading their content. If the incentive to create original, quality content disappears, society ends up losing, and the future of the Internet is at risk.”

Pay per crawl

Cloudflare is testing a new mechanism payment mechanism, pay per crawl, that enables website owners to decide whether they will permit AI crawlers to access their content, and if that access will be free or they will charge for it. The technology, now in private beta, integrates with existing web infrastructure to create a framework to enable site owners to require payment, and tell the crawler the price via an HTTP “402 payment required“ response code.

The site owner can currently set a single price for the site or choose to let certain crawlers access it at no charge, but Cloudflare expects the feature to evolve over time, perhaps to allow dynamic pricing, or charge different amounts for various types of content.

“The true potential of pay per crawl may emerge in an agentic world,” the company said in a blog post about the new feature. “What if an agentic paywall could operate at the network edge, entirely programmatically? Imagine asking your favorite deep research program to help you synthesize the latest cancer research or a legal brief, or just help you find the best restaurant in Soho — and then giving that agent a budget to spend to acquire the best and most relevant content.”

Cloudflare acts as the merchant of record for the purchases, billing the crawlers and distributing the funds to the site owners.

If the crawler doesn’t yet have a billing relationship with Cloudflare, it is blocked but receives an error message indicating that with such a relationship it might gain access to the content.

Cloudflare has invited both crawlers interested in paying for content and content owners who wish to be paid to sign up for the beta; existing enterprise customers can also contact their account executive.

A win-win

Fritz Jean-Louis, principal cybersecurity advisor at Info-Tech Research Group, sees the approach as a positive move which addresses concerns about unauthorized use of content by AI crawlers..

“By giving website owners control over how their content is accessed and used by AI crawlers, this solution empowers content creators to protect their intellectual property and potentially monetize their content more effectively,” he said. “The requirement for AI companies to disclose the purpose of their crawlers introduces a level of transparency and accountability that has been lacking in the industry, helping to build trust between content creators and AI companies.”

But he does see unresolved issues that need addressing, such as how to handle what he called “legacy” information that had already been scooped up by crawlers.

Jean-Louis favors industry-driven solutions over punitive regulations: “This move by Cloudflare could indicate a shift in the industry toward supporting a fair and sustainable digital ecosystem, balancing the needs of content creators and AI innovators: a win-win situation.”Zuckerberg announces Meta Superintelligence Labs – ComputerworldRead More