Guide · Cloudflare

Does Cloudflare block AI crawlers by default? How to check and allow them

Updated 2026-11-07 · kagentx

Short answer

Cloudflare's 2025-07-01 press release says AI crawlers are blocked by default, while the current docs (updated 2026-11-02) say Search, Agents and Training all default to Allow for new domains. We found no explanation for the difference, so trust your own dashboard: Security > Settings > AI Crawler and Agent access policies. If Search is set to Block, search crawlers cannot get in even if robots.txt allows them. To be recommended by AI, keep at least Search and Agent on Allow.

Does Cloudflare really block AI crawlers by default?

Two official Cloudflare statements do not agree and we found no explanation, so we list both. The 2025-07-01 press release says Cloudflare now blocks AI crawlers by default and asks the owner of a new domain whether to allow them, changeable at any time. The current developer documentation (page dated 2026-11-02) says that for new domains the Search, Agents and Training categories all default to Allow. Either way, the safest approach is not to guess but to look at your own dashboard; the “How do I check” section below gives the path.

SourceDateWhat it says about the default for new domains
Cloudflare press release2025-07-01Blocks AI crawlers by default; new domains are asked whether to allow them, changeable at any time
Cloudflare developer docs: AI Crawler and Agent access policiesPage dated 2026-11-02; we viewed it on 2026-11-07Search, Agents and Training all default to Allow (not blocked); can be changed later

What are these Cloudflare features?

There are many names and some were renamed, so line them up first. Every row below comes from an official Cloudflare page, each opened and checked on 2026-11-07:

Namewhat it isStatus and date
One-click AI crawler blockA switch launched in 2024-09The press release says over one million customers turned it on (as stated on 2025-07-01)
AI Crawl Control (formerly AI Audit)See which AI services crawl your site, allow or block individual crawlers, and track whether robots.txt is respectedThe 2025-08-28 blog post announced the rename from AI Audit and the move from beta to general availability; the docs say it is available on all plans
AI Crawler and Agent access policiesThree behavior-based categories: Search, Agent, TrainingCurrent name of the settings, formerly Configure AI bot policies; docs updated 2026-11-02
Pay per crawlSet a price for each AI crawler request; without payment it gets HTTP 402The docs say “closed beta” (updated 2026-07-28); you must apply
Managed robots.txtCloudflare adds AI crawler directives to your robots.txt based on your settingsAll plans; docs updated 2026-11-02

What do the Search, Agent and Training settings each block?

The current docs classify crawlers by behavior, not by company. The available actions for each category are:

CategoryOfficial definitionExamples (types in the official docs)
SearchCrawlers that collect or index content to answer questions laterSearch-type crawlers
AgentAutomated tools acting in real time on behalf of a personChat fetch bots and browser-use agents
TrainingCrawlers that gather content to train or fine-tune models, including mixed-purpose crawlers used for both training and searchTraining-type crawlers
ActionMeaning
BlockBlocks matching requests across the whole site
Block on pages with adsBlocks only on pages where Cloudflare detects ads
Allow (do not block)Does not block; choose this to allow
Disallow (Training only)Tells crawlers via robots.txt not to crawl; this is voluntary and crawlers can ignore it

There is a direct consequence for a product site that wants AI recommendations: if you set Search to Block, search crawlers such as OAI-SearchBot and PerplexityBot cannot get in, and allowing them in robots.txt does not help because the block happens at the request layer. Mixed-purpose crawlers are also affected by the Training action. The purpose of each crawler is in AI crawler user agents: a side-by-side table.

Are Pay per crawl and managed robots.txt useful to indie developers?

You can ignore Pay per crawl for now: the docs say it is in closed beta, requires an application form, and charges AI crawlers for access, with Cloudflare as the payment party. A crawler that shows payment intent gets 200, otherwise it gets an HTTP 402 containing the price. The docs also say that if WAF or Bot Management rules block a crawler, those rules take priority over charging and the crawler cannot get in.

Managed robots.txt is worth knowing about. The docs say it only changes your robots.txt when at least one category is set to Block or Disallow; when everything is allowed your own file is returned unchanged. When enabled it adds Disallow rules plus a Content Signals block, defaulting to search=yes, ai-train=no; the example lists Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and meta-externalagent. So if your robots.txt contains AI crawler sections you never wrote, this is probably where they came from:

# What a Cloudflare-managed block looks like when Training is set to Disallow (shape only, not the full text)
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

# plus a Content Signals block, default: search=yes, ai-train=no

How do I check whether my site blocks GPTBot, ClaudeBot or PerplexityBot?

  1. 1. Look at the settings in the Cloudflare dashboard

    Open your domain, go to Security > Settings, filter by Bot traffic, and open AI Crawler and Agent access policies (the path given in the docs).

  2. 2. Look at the AI Crawl Control crawler list

    In the dashboard open AI Crawl Control and the Crawlers tab to see the AI crawlers visiting your domain; the Action column offers Block. The docs say Metrics on this tab covers only the last 24 hours on some plans, and that Enterprise with Bot Management uses finer identification while the rest identify by User-Agent.

  3. 3. Read your own robots.txt

    Open yourdomain/robots.txt. If it has GPTBot or ClaudeBot sections you did not write, managed robots.txt added them. You can also enter your URL in AI crawler check to see whether each crawler is allowed or blocked in robots.txt.

  4. 4. Request your site once pretending to be a crawler with curl

    The script below requests your home page using four crawler names. A 403 or a challenge page means a rule matching on User-Agent is blocking; 200 means that kind of rule is not blocking. Note that it only tests rules that match on User-Agent, not rules that verify the source IP, because your IP is not a crawler's IP.

    # Does a rule that matches on the User-Agent string block this bot? (this does NOT test IP-based rules)
    for ua in "GPTBot" "ClaudeBot" "PerplexityBot" "OAI-SearchBot"; do
      printf "%s " "$ua"
      curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (compatible; $ua)" https://your-site.com/
    done
  5. 5. In the end, rely on server logs

    Only seeing real crawler requests and their status codes in your logs tells you for sure whether they got in. A method for PerplexityBot is in How to get Perplexity to recommend my product.

What does kagentx's own site look like?

kagentx.com is on Cloudflare (response header server: cloudflare). On 2026-11-07 we read its live robots.txt: a single general rule group that allows every crawler, disallows only a few admin pages and /api/, and has no Cloudflare-managed AI crawler section:

User-agent: *
Allow: /

# Admin / local tools — no need to index
Disallow: /admin.html
Disallow: /candidates.html
Disallow: /api/

Sitemap: https://kagentx.com/sitemap.xml

Requesting the home page with the User-Agent strings of GPTBot, ClaudeBot and PerplexityBot returned HTTP 200 for all three. We simulated this with strings from our own machine, not real crawlers (the source IP differs), so it only shows that no rule matching on User-Agent is blocking, not that real crawlers will certainly get in.

I confirmed I'm blocked. How do I allow them?

  1. 1. Change the relevant category to Allow

    To appear in AI search answers, keep at least Search and Agent on Allow (do not block). Whether to allow Training is your own decision: it determines whether content can be used for training, which is separate from appearing in search answers.

  2. 2. Check Bot Fight Mode and custom rules

    Perplexity's official page says Bot Fight Mode cannot be skipped by custom rules and may still challenge its crawlers. It suggests a custom rule matching the User-Agent plus the official IP list, with the action Skip, placed above every blocking or challenging rule.

  3. 3. Check robots.txt as well

    After removing the block, make sure no stale Disallow remains in robots.txt. How to write it is in How to write robots.txt for AI crawlers.

  4. 4. Look at the result after a while

    Confirm in the AI Crawl Control crawler list and your logs that requests are arriving. To see how this affects ChatGPT recommendations, read How to get ChatGPT to recommend my product.

For commands that read the server logs directly, without Cloudflare, see How to check your server logs for AI crawlers.

To limit crawler request speed with Cloudflare, see Will AI crawlers crawl too much? Which crawlers honor Crawl-delay.

FAQ

Will Cloudflare block AI crawlers on my website by default?

Cloudflare's 2025-07-01 press release says it blocks by default and asks when a new domain signs up; the current docs (updated 2026-11-02) say new domains default to Allow in all three categories. The two statements disagree, so look at AI Crawler and Agent access policies in your own dashboard.

Are AI Crawl Control and AI Audit the same thing?

Yes. Cloudflare's 2025-08-28 blog post says AI Audit was renamed AI Crawl Control and moved from beta to general availability.

Can I use Pay per crawl now?

The docs (updated 2026-07-28) say it is in closed beta and requires an application. It lets site owners set a price for AI crawler access; without payment the crawler gets HTTP 402.

Why does my robots.txt contain rules for GPTBot?

It may be Cloudflare's managed robots.txt. The docs say that when at least one category is set to Block or Disallow, it prepends AI crawler Disallow rules and Content Signals to your robots.txt.

How do I allow GPTBot, ClaudeBot and PerplexityBot?

In Security > Settings > AI Crawler and Agent access policies, change the relevant category to Allow, then check Bot Fight Mode and custom rules, and make sure no stale Disallow remains in robots.txt.

Sources

kagentx

Can crawlers read your pages?
Let the evaluation tell you

The evaluation checks whether crawlers can get in and whether the text is served directly, then asks ChatGPT, Perplexity and Claude (with web search on) your buyers' questions and sees whether you are mentioned. The Basic plan is $1, and results usually take 2–3 minutes.