Cloudflare draws battle lines for the agentic web with September mixed-crawler block

New defaults split AI bot traffic into Search, Agent, and Training — and pull mixed-use crawlers like Googlebot off ad-monetized pages unless operators separate them by Sept. 15.

Cloudflare gave AI crawler operators a 10-week deadline on Wednesday to split their bots into three declared categories or lose access to ad-monetized pages across roughly 20% of the web. The new defaults take effect September 15, 2026, and apply to every new domain, every new site added by existing customers, and every free-tier account that hasn’t opted out.

The taxonomy is the whole story. Cloudflare now classifies crawlers as Search, Agent, or Training, and it’ll apply the most restrictive rule any operator hasn’t distinguished away. Googlebot, Applebot, and BingBot are the obvious casualties: they bundle traditional search indexing with AI training in a single user agent, and unless their owners separate them, they get treated as Training and blocked from monetized pages.

The data behind the policy comes from Cloudflare’s one-year bot report. AI training crawls climbed from 22% of crawler requests in spring 2025 to 52% by June 2026. Mixed-use crawlers now account for over 36% of activity. Cloudflare argues this bundling has handed Google “about 2X more information than leading AI companies,” a competitive asymmetry the company frames as structural rather than incidental. Google’s counter, that Google-Extended lets publishers opt out of Gemini and Vertex training, doesn’t cover AI Overviews or AI Mode, which is precisely why Cloudflare isn’t treating it as a resolution.

Alongside the block, Cloudflare upgraded its 2025 Pay Per Crawl product into Pay Per Use, a value-based billing layer with Ceramic.ai and You.com as launch partners. The pitch leans on another internal number: over 50% of AI-crawler traffic re-fetches pages that haven’t changed.

“Now that the majority of traffic on the Internet is non-human, we must go further and act faster,” said CEO Matthew Prince.

The move echoes the 2003 CAN-SPAM moment, when infrastructure providers, not legislators, drew the operational line on what commercial traffic the network would carry. The agentic web is getting the same treatment, from the same layer.