This year, Cloudflare introduced a marketplace that enables websites to charge AI bots for scraping their content. However, the web infrastructure provider is now advocating for stricter regulations in the AI industry.
Cloudflare’s CEO, Matthew Prince, recently visited London to engage with the U.K.’s Competition and Markets Authority (CMA). He is advocating for more rigorous rules governing Google’s competition in the AI landscape, emphasizing its dominance in search.
Earlier this month, the CMA granted Google a special designation in search and advertising markets due to its “substantial and entrenched” position. This designation allows the regulator to impose more stringent rules beyond traditional searches and advertising, extending to areas such as Google’s AI Overviews, AI Mode, Discover feed, Top Stories, and News tab.
Prince believes that Cloudflare is well-positioned to offer recommendations since the company is not directly involved in the AI sector, yet maintains extensive relationships with various AI businesses.
“We’re not an AI company,” remarked Prince during his address at the Bloomberg Tech conference in London this week. “We serve as a critical infrastructure layer between AI companies and content providers, with 80% of AI companies relying on our services,” he noted.
He argues that Google should not enjoy an unfair competitive edge over other AI firms. Currently, Google leverages its web crawler not only for its search engine but also for its AI features, presenting an advantage that other companies lack. Prince referred specifically to Googlebot, which crawls content for search, including AI functionalities like AI Overviews. A Google spokesperson, Ned Adriance, mentioned that site owners can opt out of having their content used for AI training with Google Extended, without affecting their search visibility. However, many media businesses would likely prefer to completely withdraw from any AI implications, making this argument significant.
“Google insists on its right to access all content globally, without compensation, based on its 27-year legacy,” Prince explained. “They argue they can utilize the same crawler they use for search to enhance their AI systems. The dilemma is that opting out of one service requires opting out of both,” he elaborated.
Techcrunch event
San Francisco
|
October 27-29, 2025
This scenario is particularly unfeasible for those in the media sector, where losing search visibility can equate to a 20% revenue loss, according to the executive.
“It gets worse: blocking Google’s crawler also disables their ad safety team, effectively nullifying advertisements on all platforms, which is simply not an option,” Prince asserted.
By bundling its crawler services, Google gains access to content that competitors like Anthropic, OpenAI, and Perplexity must pay for.
“The situation essentially gives Google a monopoly on content access,” Prince stated.
To rectify this imbalance, Prince believes fostering competition is crucial—allowing potentially thousands of AI companies to acquire content from numerous media and small business entities. He lauded the CMA’s decision to flag Google for further regulatory scrutiny, noting it reflects an awareness of the distinct advantages Google holds.
Cloudflare has provided the CMA with data illustrating the mechanics of Google’s crawler, highlighting the near impossibility of replicating Google’s success.
Prince’s sentiments echo those of Neil Vogel, CEO of People, Inc., the largest digital and print publisher in the U.S. In a recent interview, Vogel classified Google as a “bad actor,” noting that media companies face pressure to allow Google access for AI content due to the intertwined nature of the crawling systems.
Vogel’s company has adopted Cloudflare’s solution to block unpaid AI crawlers, asserting that it has effectively initiated negotiations with several leading large language model providers.
Updated with Google comment.
