The main AI crawlers, checked one by one
Every row below was checked by opening the official page on 2026-11-05. Rows we could not check say “Not verified”, and we do not fill them from third-party directories. “Use” follows the official wording in three groups: training (collecting content to train models), search (building an index for AI search) and user-triggered (opening a page live when a user asks).
| Crawler / token | Company | Purpose | Obeys robots.txt? (official wording) | User-Agent per the docs | Official docs | Checked on |
|---|---|---|---|---|---|---|
| GPTBot | OpenAI | Training | Controlled with robots.txt; blocking it signals the content should not be used to train foundation models | GPTBot/1.4 | OpenAI | 2026-11-05 |
| OAI-SearchBot | OpenAI | Search | Controlled with robots.txt; allowing it is recommended, and blocked sites do not appear in ChatGPT search answers | OAI-SearchBot/1.4 | OpenAI | 2026-11-05 |
| ChatGPT-User | OpenAI | User-triggered | The docs say: user-initiated visit, robots.txt rules may not apply | ChatGPT-User/1.0 | OpenAI | 2026-11-05 |
| ClaudeBot | Anthropic | Training | Obeys (Anthropic: its crawlers honor robots.txt) | Full string not listed on the official page | Anthropic | 2026-11-05 |
| Claude-User | Anthropic | User-triggered | Obeys (same statement, covers all three) | Full string not listed on the official page | Anthropic | 2026-11-05 |
| Claude-SearchBot | Anthropic | Search | Obeys (same statement, covers all three) | Full string not listed on the official page | Anthropic | 2026-11-05 |
| PerplexityBot | Perplexity | Search | Can be managed with robots.txt; allowing it is recommended; the docs say it is not used for training foundation models | PerplexityBot/1.0 | Perplexity | 2026-11-05 |
| Perplexity-User | Perplexity | User-triggered | The docs say: because the user requested the visit, it generally ignores robots.txt | Perplexity-User/1.0 | Perplexity | 2026-11-05 |
| Google-Extended | Training and grounding (control token) | Only a token in robots.txt with no separate User-Agent; does not affect Search inclusion and is not a ranking signal | None (crawling uses Google's existing crawlers) | 2026-11-05 | ||
| Googlebot | Traditional search | Google: automatic crawls always obey robots.txt | The official page lists smartphone and desktop strings | 2026-11-05 | ||
| Applebot | Apple | Search (data may be used for training) | Obeys; follows Googlebot rules when there are no Applebot rules; the iTMS user agent does not | Full string not retrieved from the official page | Apple | 2026-11-05 |
| Applebot-Extended | Apple | Training (control token) | Does not crawl pages; only decides whether content Applebot fetched can be used for training; blocked pages can still appear in search | None (not a crawler) | Apple | 2026-11-05 |
| CCBot | Common Crawl | Open web dataset | The docs give a robots.txt blocking rule but do not say outright that it obeys | CCBot/2.0 | Common Crawl | 2026-11-05 |
| Bingbot | Microsoft | Traditional search | Not verified | Not verified | Microsoft (page text could not be retrieved) | Not verified |
| Bytespider | ByteDance | Not verified | Not verified: no official ByteDance documentation was found and third-party claims disagree, so we draw no conclusion | Not verified | None | Not verified |
Which ones are crawlers and which are only robots.txt tokens?
Google-Extended and Applebot-Extended are not crawlers that visit your server. Google's documentation says Google-Extended has no separate User-Agent; Apple's says Applebot-Extended does not crawl webpages. They are only control tokens you write in robots.txt, so you will not find those two names in server logs; search for Googlebot or Applebot instead.
How do I confirm a User-Agent is genuine?
Anyone can fake a User-Agent string. Google's documentation notes that genuine common crawlers generally come from published IP ranges whose reverse DNS matches; OpenAI and Perplexity also publish IP lists (one JSON per crawler at OpenAI, two at Perplexity), and Common Crawl's page warns that some crawlers falsely use the CCBot name. When writing firewall rules, check against the IP lists and do not trust the string alone.
What to read next
This table only answers “who is it and what does it control”. For how to write robots.txt, what happens when you block, and three templates, see How to write robots.txt for AI crawlers; to check what your own site allows right now, use AI crawler check. For how each assistant works, see How to get ChatGPT to recommend my product and How to get cited by Perplexity.
FAQ
What is the difference between GPTBot and OAI-SearchBot?
OpenAI's documentation says GPTBot collects content that may be used to train foundation models, while OAI-SearchBot lets sites appear in ChatGPT search. Blocking GPTBot does not opt you out of search.
What is Google-Extended's User-Agent?
There is none. Google's documentation says it has no separate HTTP User-Agent; crawling uses Google's existing crawlers, and Google-Extended is only a control token in robots.txt.
Which AI crawlers ignore robots.txt?
Two are stated officially: Perplexity says Perplexity-User generally ignores robots.txt, and OpenAI says robots.txt rules may not apply to ChatGPT-User visits. Both are user-triggered visits. We found no official statement on whether Bytespider obeys, so it is not verified.
Is Bingbot an AI crawler?
It is Bing search's crawler. We could not retrieve the text of the Bing Webmaster Tools help page, so its specifics are marked not verified here.
Will this table go out of date?
Yes. Crawler names and version numbers change. Every row carries its check date; go back to the official page before relying on it.
Sources
- OpenAI · Overview of OpenAI Crawlers （Checked 2026-11-05）
- Anthropic · Does Anthropic crawl data from the web? （Checked 2026-11-05）
- Perplexity · Perplexity Crawlers （Checked 2026-11-05; the page carries no date）
- Google Search Central · Google’s common crawlers （Checked 2026-11-05; the page carries no date）
- Apple · About Applebot （Checked 2026-11-05）
- Common Crawl · CCBot （Checked 2026-11-05）
- kagentx evaluations: the data on the case study pages; each example gives its date.