robots.txt &
AI crawler check
Enter a URL to see whether 16 AI and search crawlers, including GPTBot, ClaudeBot, PerplexityBot and Google-Extended, can read your page: a verdict for each, the robots.txt line that decides it, and how to fix it.
Free, no sign-up. Only the site's public files are read; the URL you enter is not stored.
Which crawlers, what blocking does
Four kinds by purpose. AI search crawlers and the ones that open links for users decide whether you show up in AI answers; training crawlers only affect model training.
| Crawler | Purpose | what blocking does | Source |
|---|---|---|---|
| OAI-SearchBot OpenAI · ChatGPT search | AI search | Builds the AI's search index. Block it and that AI's answers will rarely cite you. | Docs |
| ChatGPT-User OpenAI · ChatGPT | Opens links for users | Fetches a page when someone pastes your link into the AI or the AI opens it while answering. Block it and the AI can't open your page. | Docs |
| GPTBot OpenAI · OpenAI models | Model training | Collects pages to train models. Blocking it does not keep you out of AI answers; it's your call. | Docs |
| Claude-SearchBot Anthropic · Claude search | AI search | Builds the AI's search index. Block it and that AI's answers will rarely cite you. | Docs |
| Claude-User Anthropic · Claude | Opens links for users | Fetches a page when someone pastes your link into the AI or the AI opens it while answering. Block it and the AI can't open your page. | Docs |
| ClaudeBot Anthropic · Claude models | Model training | Collects pages to train models. Blocking it does not keep you out of AI answers; it's your call. | Docs |
| PerplexityBot Perplexity · Perplexity search | AI search | Builds the AI's search index. Block it and that AI's answers will rarely cite you. | Docs |
| Perplexity-User Perplexity · Perplexity | Opens links for users | Fetches a page when someone pastes your link into the AI or the AI opens it while answering. Block it and the AI can't open your page. | Docs |
| Google-Extended Google · Gemini models | Model training | Collects pages to train models. Blocking it does not keep you out of AI answers; it's your call. | Docs |
| Googlebot Google · Google Search, AI Overviews | Search engine | Classic search engines; AI answers lean on their results too (Bing feeds ChatGPT search and Copilot, Google has AI Overviews). | Docs |
| Bingbot Microsoft · Bing, Copilot | Search engine | Classic search engines; AI answers lean on their results too (Bing feeds ChatGPT search and Copilot, Google has AI Overviews). | Docs |
| Applebot-Extended Apple · Apple Intelligence models | Model training | Collects pages to train models. Blocking it does not keep you out of AI answers; it's your call. | Docs |
| Meta-ExternalAgent Meta · Meta AI models | Model training | Collects pages to train models. Blocking it does not keep you out of AI answers; it's your call. | Docs |
| CCBot Common Crawl · Common Crawl dataset | Model training | Collects pages to train models. Blocking it does not keep you out of AI answers; it's your call. | Docs |
| Amazonbot Amazon · Alexa, Rufus | AI search | Builds the AI's search index. Block it and that AI's answers will rarely cite you. | Docs |
| DuckAssistBot DuckDuckGo · DuckAssist | Opens links for users | Fetches a page when someone pastes your link into the AI or the AI opens it while answering. Block it and the AI can't open your page. | Docs |
How is allowed or blocked decided?
By the robots.txt standard (RFC 9309): a crawler first looks for a group with its own name and only falls back to User-agent: * when there is none; within the group the rule with the longest matching path wins, and Allow wins a tie. The results table shows which line decided.
What if robots.txt can't be opened?
A 404 or other 4xx counts as no rules: everything is allowed. A 5xx, 429, timeout or no connection makes crawlers such as Google's and OpenAI's treat the whole site as off limits until it opens again.
Should I block training crawlers?
Your call. Blocking training crawlers such as GPTBot, ClaudeBot and Google-Extended does not stop ChatGPT search, Claude or Perplexity from citing you; allowing them lets models learn more about your product.
robots.txt allows everything, but AI still can't read my site?
robots.txt is only the first gate. A CDN's "block AI bots" switch, firewall rules and bot challenges stop requests outright, and if your page text only appears after JavaScript runs, crawlers read little of it.
Is this tool free?
Yes, free and no sign-up. kagentx's evaluation is paid: it actually asks AI products and checks whether the answers name you and whom they cite.
Crawlers can get in.
Does AI recommend you?
An evaluation actually asks AI products the questions people ask in your category and checks whether the answers name you. Basic is $1, usually ready in 2–3 minutes.