Cloudflare AI Crawl Control: is it blocking AI crawlers on your site?

Cloudflare is blocking AI crawlers by default on some sites and not others, depending on when the domain joined, whether it shows ads and what was switched on before September 15, 2026.

On July 1, 2025 Cloudflare started asking every new domain at sign-up whether to allow AI crawlers, with blocking as the starting point. A year later it split that into three behaviors (Search, Agent and Training), each set to Allow, Block on pages with ads or Block. Then Cloudflare's September 15, 2026 update gave Training a fourth option, Disallow AI Training, began retiring the old Block AI bots toggle and migrated existing settings, so a domain that had the old block on now has Search on Allow, Training on Disallow AI Training and Agent on Block on pages with ads. New domains answer one question about ads at onboarding: no ads, nothing is blocked; ads, and they start where those migrated domains ended up.

So search crawlers are allowed almost everywhere (Cloudflare says fewer than 1% of its sites block Search bots), while 17% of its sites block training in some way.

Cloudflare's managed robots.txt is a set of robots.txt rules Cloudflare writes for you, and the Content Signals Policy (added in September 2025) is the block of comments on top that defines three signals: search, ai-input and ai-train.

Per Cloudflare's robots.txt setting docs, the managed version disallows training crawlers including CCBot, GPTBot, ClaudeBot and Google-Extended, adds a Content-Signal line of search=yes, ai-train=no, and is prepended to your own file, so your robots.txt can say things you never typed. A signal is a preference, not a lock, and Cloudflare itself says to pair signals with WAF rules.

Since September 15, 2026 Bot Preference Sync is replacing it, writing whatever your Search, Training and Agent settings say into robots.txt, and it's on by default for new customers. Read the lines between Cloudflare's BEGIN and END comments in yourdomain.com/robots.txt, because that's what CCBot, Common Crawl's crawler, reads first.

To allow AI search crawlers in Cloudflare, set Search to Allow, check each search crawler shows Allow in AI Crawl Control, and make sure no rule higher up turns them away:

1. Open the domain, go to Security > Settings, filter by Bot traffic and open Configure AI bot policies. Set Search to Allow, and Agent too if you want assistants fetching pages live when a user asks.
2. In AI Crawl Control, open the Crawlers tab and check that OAI-SearchBot (how OAI-SearchBot differs from GPTBot), Claude-SearchBot and PerplexityBot say Allow in the Action column.
3. Check CCBot too if you want Common Crawl to keep saving your pages: Cloudflare's robots.txt template files it with the training crawlers, so Training on anything but Allow is the first suspect.
4. Go to Security > Security rules, filter by Custom rules and read every rule above the one called AI Crawl Control, looking for country blocks, datacenter challenges and bad-bot lists. A Skip rule at the top for the verified categories you want (AI Search, AI Assistant) gets those bots past them.
5. Check Bot Fight Mode. Custom rules can't skip it, so if it's challenging a crawler you want, turn it off (Super Bot Fight Mode on paid plans does take Skip rules).

To block AI training bots in Cloudflare and keep AI search, set Training to Disallow AI Training, leave Search on Allow, and do NOT pick Block.

Since September 15, 2026, Block also catches mixed-use crawlers, and Cloudflare's post is blunt that it "will stop Applebot, Bingbot, and Googlebot from reaching your site", search included.

Disallow AI Training splits the two jobs: it publishes the no-training preference in robots.txt (Google-Extended and Applebot-Extended carry it for Google and Apple), keeps Googlebot, Bingbot and Applebot crawling for search, and blocks the training-only crawlers from OpenAI, Anthropic, Meta and Amazon. Google says Google-Extended doesn't affect Search inclusion or rankings. Bing is the gap: Microsoft is targeting early 2027 to honor a robots.txt no-training rule, and until then its opt-out is the NOARCHIVE meta tag.

To keep CCBot while turning away the model companies' own crawlers, leave Training on Allow and block GPTBot, ClaudeBot and the rest by name in AI Crawl Control.

Cloudflare Pay Per Crawl lets a site charge AI crawlers a set price per request, and Cloudflare's Pay Per Crawl docs still list it as a closed beta.

It launched on July 1, 2025: a crawler you charge either sends payment intent and gets the page or gets HTTP 402 Payment Required with the price, and Cloudflare is merchant of record. A crawler with no billing relationship can't pay, which Cloudflare itself calls the functional equivalent of a block, and firewall rules run first, so a blocked crawler never sees a price. On September 30, 2026 Cloudflare added a Pay Per Use beta that pays per reported use of content instead of per fetch.

To check if Cloudflare is blocking AI crawlers, look at what the bots were actually served, because the settings page only tells you what you asked for.

Cloudflare's side is in AI Crawl Control's Metrics tab and the Events tab under Security > Analytics. The bot's side is the part most people never see.

Check it free with Crawl Record, our Common Crawl checker: it reads the response Common Crawl's bot saved from your site and names the blocker, whether that's Cloudflare's block page, error 1020 (a firewall rule) or a challenge.

Here's a real one. On July 12, 2026 CCBot asked for the home page of getseedsrighthere.com, a store whose robots.txt let CCBot in, and got HTTP 403 with Cloudflare's standard block page and Ray ID. Crawl Record found 403s on 15 of the site's URLs across the July and August crawls; by September the block was gone and Common Crawl saved 428 pages. robots.txt said yes the whole time. The firewall said no.

Our AI visibility checker sends requests with the user agents of OAI-SearchBot, ChatGPT-User, the Claude bots and CCBot from our server, the fastest way to see if Cloudflare is blocking ChatGPT's crawlers today. One caveat, because evidence beats conjecture: a rule keyed on the user agent treats our requests like the real bot, but a rule keyed on Cloudflare's verified bot list may not. A Common Crawl record is what the real CCBot got.

What most people overlook about Cloudflare AI Crawl Control is that the firewall decides before robots.txt is read, so a bot your robots.txt allows can still be turned away.

A crawler normally asks for robots.txt first, and that request goes through Cloudflare's edge like any other: if a rule blocks the bot, it gets a 403 for robots.txt and every page after it (Cloudflare's Directives tab says as much: check upstream WAF rules when robots.txt requests fail).

It cuts the other way too. robots.txt is an honor system, and OpenAI and Perplexity both say it may not apply to fetches a user asked for. robots.txt says what a polite bot WILL take; the firewall decides what any bot CAN get.

In more than 20 years of SEO I've found indexing trouble is usually an Authority problem, not a technical one. This is the exception.

Here are quick answers on Cloudflare AI Crawl Control.

Does Cloudflare block Googlebot?

Cloudflare does not block Googlebot by default, but since September 15, 2026 it will if you set Training to Block (or Block on pages with ads, on those pages), because Googlebot is a mixed-use crawler. Disallow AI Training keeps it crawling for search.

Will blocking AI bots in Cloudflare hurt SEO?

Blocking AI bots in Cloudflare won't touch your Google rankings if you only block training-only crawlers, but blocking Search crawlers takes you out of the indexes AI assistants answer from, and Training on Block now takes Googlebot and Bingbot out too. It's just SEO.

Does robots.txt override Cloudflare's AI bot block?

No, robots.txt does not override Cloudflare's AI bot block, because the block happens at Cloudflare's edge before the bot can read the file. Change the setting or rule that stops the crawler, then check again.

Is Cloudflare blocking AI agents?

Cloudflare is blocking AI agents on pages that show ads for a lot of sites, because that's the Agent default for new ad-supported domains and every domain that had the old block on. It matters for ChatGPT-User, which OpenAI says may not follow robots.txt.

Why does Google Search Console flag my Cloudflare robots.txt?

Google Search Console can flag the Content-Signal lines in a Cloudflare robots.txt as syntax it doesn't understand, and Cloudflare says it has seen no effect on crawling rates or SEO from those warnings.