What is each AI crawler for?
A company usually has three kinds of crawler: one that collects pages to train models, one that builds the index for AI search, and one that opens a page live when a user asks. Blocking each has a very different effect, so robots.txt has to be written by crawler name. The table follows the 16 crawlers checked by the kagentx AI crawler check; the statements come from each company's official documentation, each checked on 2026-11-04.
| Crawler | Company | Purpose | Obeys robots.txt? (what the docs say) | What blocking does |
|---|---|---|---|---|
| OAI-SearchBot | OpenAI | Search index | OpenAI's docs recommend allowing it in robots.txt | OpenAI says a blocked site will not appear in ChatGPT search answers |
| ChatGPT-User | OpenAI | User-triggered | OpenAI states this is a user-initiated visit, so robots.txt rules may not apply | OpenAI says it does not decide whether content appears in search; that is OAI-SearchBot's job |
| GPTBot | OpenAI | Training | OpenAI says to control it with the GPTBot token in robots.txt | Signals that your content should not be used to train OpenAI's foundation models |
| ClaudeBot | Anthropic | Training | Anthropic says its crawlers obey robots.txt | Signals that the site's future content should be left out of Anthropic's model training sets |
| Claude-SearchBot | Anthropic | Search index | Same as above | Anthropic says your content will not be indexed for search, which may reduce your visibility in Claude search results |
| Claude-User | Anthropic | User-triggered | Same as above | Anthropic says the system cannot fetch your content when a user asks, which may reduce visibility |
| PerplexityBot | Perplexity | Search index | Perplexity recommends allowing it | The docs do not spell out the consequence; its purpose is to get sites into Perplexity search results, and Perplexity says it is not used to train foundation models |
| Perplexity-User | Perplexity | User-triggered | Perplexity states that because a user requested the fetch, it generally ignores robots.txt | robots.txt cannot stop it; to block it you have to use a firewall |
| Google-Extended | Training | Google says it is only a token in robots.txt, with no separate user agent | Google says it does not affect inclusion in Google Search and is not a ranking signal; it governs Gemini model training | |
| Googlebot | Classic search engine | Google says its common crawlers always obey robots.txt when crawling automatically | Not indexed by Google Search; AI Overviews rely on it too | |
| Bingbot | Microsoft | Classic search engine | Not verified (the official page's text could not be retrieved this time) | Not indexed by Bing (inferred from its purpose); Copilot uses Bing results |
| Applebot-Extended | Apple | Training | Apple says it does not crawl pages; it only decides whether content fetched by Applebot may be used for training | Apple says a blocked page can still appear in search results; it is just not used to train models such as Apple Intelligence |
| Meta-ExternalAgent | Meta | Training | The official page does not say; it only tells site owners to add a Disallow rule to block it | As the official page describes: after the rule is added it stops crawling |
| CCBot | Common Crawl | Public crawl dataset | Common Crawl says a Disallow in robots.txt blocks it | No longer crawled by Common Crawl |
| Amazonbot | Amazon | Improves products; may be used for training | Amazon says it obeys robots.txt and changes take about 24 hours | No longer crawled; Amazon also has Amzn-SearchBot (not used for training) and Amzn-User (user-triggered, may not follow all robots.txt rules) |
| DuckAssistBot | DuckDuckGo | User-triggered | DuckDuckGo says robots.txt can block it, taking about 72 hours | DuckDuckGo says it does not affect organic search rankings and the data is not used for training |
The table comes down to one point: blocking a training crawler only affects whether your content is used to train models; blocking a search or user-triggered crawler affects whether you appear in AI answers. If you want AI to recommend you, do not block the latter two.
How do I block GPTBot but still let ChatGPT search cite me?
OpenAI gives training and search to two different crawlers: GPTBot for training and OAI-SearchBot for ChatGPT search. So you can block GPTBot alone and explicitly allow the other two:
User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: /
OpenAI's docs say search results can take about 24 hours to catch up with a robots.txt change. Anthropic also separates them: ClaudeBot handles training, and Claude-SearchBot and Claude-User handle search, so write them separately.
Three robots.txt files you can copy as they are
Replace your-site.com with your domain and put the file at /robots.txt in the site root. The three differ only in what they block. Crawler names and roles change, so check the current list with the AI crawler check before using one.
A. Allow everything: you want every AI to be able to read you
This is the best fit for a site that wants AI to recommend it. Having no rules at all also allows everything; writing it out makes the intent clear and makes it easier to add rules later.
# Allow every crawler, including AI search and AI assistants User-agent: * Allow: / Sitemap: https://your-site.com/sitemap.xml
B. Block training only: you don't want to be used for training but want to appear in AI answers
Block the training crawlers as one group and leave the search and user-triggered ones alone. Amazon's documentation says Amazonbot may be used to train Amazon's AI models; if you want to block that, add it to this group.
# Block crawlers that collect pages to train models User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: Meta-ExternalAgent User-agent: CCBot Disallow: / # Everyone else, including AI search crawlers, stays allowed User-agent: * Allow: / Sitemap: https://your-site.com/sitemap.xml
C. Block everything: you don't want any AI product to use your content
This also blocks AI search and AI assistants, at the cost of not appearing in AI answers. Because Perplexity-User and ChatGPT-User are user-initiated, their companies say they may not obey robots.txt, so a full block there takes a firewall or CDN rule based on the IPs they publish. Googlebot and Bingbot are left out: blocking them would remove you from classic search too.
# Block AI training, AI search and AI assistants User-agent: GPTBot User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: ClaudeBot User-agent: Claude-SearchBot User-agent: Claude-User User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Google-Extended User-agent: Applebot-Extended User-agent: Meta-ExternalAgent User-agent: CCBot User-agent: Amazonbot User-agent: DuckAssistBot Disallow: / # Classic search engines stay allowed User-agent: * Allow: /
One trap when writing robots.txt: a crawler obeys only the group that names it and no longer reads the * group. In the file below, OAI-SearchBot sees only its own Allow, so /admin/ no longer applies to it; repeat the rule in its group:
User-agent: * Disallow: /admin/ User-agent: OAI-SearchBot Allow: /
Will Cloudflare's AI crawler blocking lock me out?
It can, regardless of what robots.txt says. Cloudflare's settings block at the request layer; they do not edit your robots.txt:
- Cloudflare's announcement of 2025-07-01 said that newly added domains are asked whether to allow AI crawlers; it also noted that more than a million customers had turned on the one-click AI crawler block launched in 2024-09.
- The current docs set policy by behavior in three groups: Search, Agent and Training. The actions are Block, Block on pages with ads and Allow, with Disallow (Training only) for training. The docs say new domains default to Allow for all three, and you can change that later.
- Search is its own group, and Block applies to matching requests across the whole zone. So setting Search to Block keeps search crawlers such as OAI-SearchBot out, and allowing them in robots.txt does not help.
Your site's setting is in the Cloudflare dashboard: Security → Settings, filter by Bot traffic, and find AI Crawler and Agent access policies (the name used in the docs). Run the AI crawler check once: if robots.txt says allowed but fetching is actually refused, the CDN or firewall is the likely cause.
How do I confirm the change took effect?
1. Look at the file itself
Open your-site.com/robots.txt in a browser and confirm it is what you just wrote, returns 200, and is not an old cached page or a 404.
2. Check each crawler with the tool
Enter the URL in the AI crawler check; it tells you per crawler whether it is allowed or blocked, and which line decides it.
3. Then look at the firewall and CDN
Request the home page with a crawler's User-Agent; the status code should be 200:
curl -s -o /dev/null -w "%{http_code}" -A "OAI-SearchBot/1.3" https://your-site.com/This only changes the User-Agent and cannot imitate a crawler's real IP. Some CDNs identify crawlers by IP, so the result may differ from the real crawler's.
4. Wait for the time the docs state
OpenAI, Perplexity and Amazon say about 24 hours; DuckDuckGo says about 72 hours. After that, ask the AI the same buyer questions a few times and see whether anything changed. For how to ask, see Why ChatGPT doesn't recommend my product and How to get cited by Perplexity.
robots.txt only controls whether a crawler may fetch, not what it gets. If your body text only appears after JavaScript runs, allowing the crawler still means it reads nothing; see Do AI crawlers render JavaScript. To give AI a site summary, see What is llms.txt.
FAQ
Can robots.txt stop every AI crawler?
No. robots.txt is a request, not enforcement. Perplexity states that Perplexity-User generally ignores it, OpenAI says robots.txt rules may not apply to ChatGPT-User visits, and Meta says Meta-ExternalFetcher may bypass it. A full block takes a firewall or CDN rule.
Will blocking Google-Extended drop me out of Google Search?
No. Google's documentation says Google-Extended does not affect a site's inclusion in Google Search and is not a ranking signal; it only governs Gemini model training.
If I block GPTBot, will ChatGPT stop recommending me?
No. GPTBot only governs training; ChatGPT search uses OAI-SearchBot. OpenAI says only sites that block OAI-SearchBot are left out of ChatGPT search answers.
How long does a robots.txt change take to apply?
The docs of OpenAI, Perplexity and Amazon say about 24 hours; DuckDuckGo says about 72 hours. The documentation from Anthropic, Google and Apple gives no specific time.
Should I allow everything or block training only?
It is a trade-off and your call. Allowing the search and user-triggered crawlers is what lets AI cite you; blocking the training crawlers only affects whether your content is used to train models. If you want AI recommendations, at least do not block the search crawlers.
Sources
- OpenAI · Overview of OpenAI Crawlers （checked 2026-11-04）
- Perplexity · Perplexity Crawlers （checked 2026-11-04）
- Anthropic · Does Anthropic crawl data from the web, and how can site owners block the crawler? （checked 2026-11-04）
- Google · Google’s common crawlers （checked 2026-11-04）
- Apple · About Applebot （checked 2026-11-04）
- Meta · Meta Web Crawlers （checked 2026-11-04）
- Amazon · Amazonbot （checked 2026-11-04）
- DuckDuckGo · DuckAssistBot （checked 2026-11-04）
- Common Crawl · CCBot （checked 2026-11-04）
- Cloudflare · Cloudflare Just Changed How AI Crawlers Scrape the Internet-at-Large （Published 2025-07-01, checked 2026-11-04）
- Cloudflare · AI Crawler and Agent access policies （checked 2026-11-04）
- kagentx evaluations: the data on the case study pages; each example gives its date.