Although AI has many useful applications including conversational chatbots, tools for the education sector, and generation of multimedia content, one capability in particular is a double-edged sword. That is the ability to procure information from online sources. To tackle this problem, AI models typically crawl the internet to build their knowledgebase and provide answers based on that. The obvious drawback of this situation is that the owners of this website are not paid for the crawled content and they don"t get visitors either since the AI chatbot serves the required content directly to customers. Now, Cloudflare is experimenting with a solution to this problem.
In a blog post, Cloudflare says that while customers already have the option to restrict AI crawlers completely or to allow them free access, it is now working to implement a third way: make AI agents pay for the content they crawl. It is doing so by resurrecting the HTTP response code 402, which stands for payment required. Cloudflare itself will be responsible for building the technical infrastructure for this implementation and acting as the Merchant of Record.
Under this design, website owners will have the option to define a domain-wide, per-request rate. They also have the option to define a custom configuration where certain crawlers have completely free access through a mutually beneficial content partnership while all others are charged per crawl. Cloudflare has also defined security policies, payment headers, and access methodologies that developers can implement to leverage this solution. Content creators can configure their payment details through their Cloudflare account.
Cloudflare has emphasized that this is a fairly rudimentary solution right now and it expects the design to evolve over time. Enhancements in the future may include granular control for access and pricing, along with an "agentic paywall [that] could operate entirely". That said, it is important to note that this is a private beta right now and those interested can gain access to it via this portal or by contacting your Cloudflare account executive.