New Cloudflare AI Crawler Rules May Block Google Search

Cloudflare has changed how AI crawlers are managed for every customer on its network, and one detail deserves your attention before September 15: depending on your settings, the new rules can end up blocking Googlebot.

On July 1, 2026, Cloudflare replaced its single "Block AI bots" toggle with three separately controllable categories, available to all customers including the free tier [1]. Search covers crawlers that index your content to answer questions and send referrals. Agent covers bots acting live on a person's behalf, like a chat assistant fetching your page mid-conversation. Training covers crawlers collecting content to train models. For each, you can block everywhere, block only on pages showing ads, or allow.

On September 15, 2026, new defaults take effect for domains newly added to Cloudflare: Training and Agent are blocked by default on pages that display ads, while Search stays allowed [2]. This continues the direction Cloudflare set in July 2025, when it became the first major infrastructure provider to block AI crawlers by default for new domains [3]. Cloudflare sits in front of roughly one fifth of the web, so its defaults effectively set policy for a large slice of the internet [4].

The Googlebot Trap

Read that again if you run client sites. A setting that sounds like "do not let AI train on my content" can, after September 15, also mean "do not let Google crawl my site." Cloudflare is giving owners the chance to confirm their preference ahead of the date, but only to those who go looking. Plenty of site owners flipped Block AI Bots on during the past year and have not thought about it since. Those are the sites at risk.

What to Actually Do

Log in to Cloudflare, open your domain, and find the AI Crawl Control settings under Security. Then make three deliberate choices. If you want to appear in AI answers and keep receiving search referrals, Search should be allowed. Agent is worth allowing if you want assistants to be able to pull your pages live when users ask about you, which is increasingly how buying research happens. Training is the genuine judgment call: it is your content and your compensation question, and blocking it is a defensible choice, but after September 15 verify that choice is not quietly extending to Googlebot.

If you manage multiple client zones, this is a fifteen-minute sweep per account that prevents a very bad October conversation about why organic traffic fell off a cliff.

Blocked crawlers are only one of the ways a site goes invisible to AI engines. Robots rules, JavaScript rendering, and forgotten noindex tags all do the same damage more quietly, and we published a ten-minute inspection covering all of them: Can ChatGPT Actually See Your Website? For a one-click version of the check, the free AI Citability Scorer grades whether AI engines can read and cite your page.