How does Claude decide who to recommend?
Anthropic's Help Center (checked 2026-11-04) says Claude searches the web when it helps, and users can ask it to search directly; every answer that used search comes with citations so you can check the sources. So, as with ChatGPT and Perplexity, the products Claude recommends mostly come from the pages it found and read for that answer.
The official documentation does not say which search engine Claude's web search uses, so we do not guess. There are two things you can do: make sure Claude's crawlers can read you, and get yourself into the kind of pages it cites.
What do Anthropic's three crawlers each do?
| Crawler | What it does (Anthropic's wording) | What happens if you block it |
|---|---|---|
| Claude-SearchBot | Indexes content to improve the relevance and accuracy of search results | The documentation says blocking it may reduce your site's visibility and accuracy in users' search results |
| Claude-User | Fetches a page when a Claude user asks a question that needs it | The documentation says blocking it may reduce your site's visibility in user-directed web search |
| ClaudeBot | Collects web content that may be used for model training | Signals that your site's future content should be excluded from training data; the documentation does not say it affects citations in answers |
All three crawlers honor do-not-crawl directives in robots.txt, and also support the non-standard Crawl-delay. Anthropic publishes the crawler IP ranges (claude.com/crawling/bots.json), but warns that blocking by IP may not be reliable; opting out is done through robots.txt.
What are the steps?
1. Allow Claude-SearchBot and Claude-User
Enter your URL in the AI crawler check and see whether these two show as allowed. If they are blocked, add this to the very top of robots.txt:
User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: /
Whether to allow ClaudeBot is a separate decision, because it is tied to training. If you want out of training, block only that one and keep the two above. That is the reason Anthropic's documentation separates the three crawlers.
2. Then check your firewall and CDN
Even when robots.txt allows them, a CDN's "block AI crawlers" switch or bot challenge can still drop the request, and those rules ignore robots.txt. Allow Anthropic's published IP ranges, or request your homepage with the crawler's User-Agent and see whether it returns 200 or 403:
curl -s -o /dev/null -w "%{http_code}" -A "Claude-SearchBot" https://your-site.com/3. Put the body text straight in the HTML
Crawlers read the raw HTML. Run the line below, replacing the quoted text with a sentence from your homepage. If it prints 0, the body text only appears after scripts run:
curl -s https://your-site.com/ | grep -c "a sentence from your homepage"
4. Ask Claude each buyer question 3 times, in the buyer's own words
Write 5–10 buyer questions without your brand name, ask each one 3 times in Claude with web search on (a new conversation each time), and record: whether you are mentioned, who is recommended, and which URLs are cited. The answer to one question differs every time, so what matters is how many of the three times you appear.
5. Fill the gaps in the pages Claude cites
Sort the cited URLs by type: list articles, tutorial blogs, comparison pages, forum threads. Where a page lists similar products but not yours, write to the author and explain how you differ from the listed products and what you cost; write the missing pages yourself, with the buyer's question as the heading and the answer in the first paragraph.
6. Add an llms.txt (lowest priority)
Write a draft with the llms.txt generator. Anthropic's documentation does not promise to read it, so it comes after the first five steps.
What kind of pages does Claude cite?
On 2026-10-22 we evaluated MochiStamp (Japanese market): we put buyer questions to Claude (Claude Sonnet 5.5, web search on) 3 times and it mentioned the site 0 times. The AI read 10 cited pages, 10 of which listed similar products, and 0 mentioned the site; the most cited were tutorials and blogs in Japanese such as noripon.blog, ai-rise.net, note.com.
On 2026-10-26 we evaluated GammaSlide: asked Claude 21 times, it mentioned the site 0 times, and the most recommended were Gamma, Plus AI, Beautiful.ai. The AI read 28 cited pages, 19 of which listed similar products, and 0 mentioned the site. Both sites passed the crawler checks; what held them back was that the cited pages did not include them. Full data in the MochiStamp case study and the GammaSlide case study.
For a fuller checklist, see the AI visibility audit checklist; for the equivalent ChatGPT troubleshooting, see Why doesn't ChatGPT recommend my product?
FAQ
Will blocking ClaudeBot stop Claude recommending me?
Anthropic's documentation says blocking ClaudeBot signals that your site's future content should not go into training data. The documentation notes that it is Claude-SearchBot and Claude-User that affect search visibility, so to opt out of training while staying citable, block ClaudeBot and allow the other two.
Which search engine does Claude's web search use?
Anthropic's help documentation does not say, so we do not draw a conclusion. What we can confirm is that Claude searches when it helps and its answers carry citations.
How long does a robots.txt change take to apply?
Anthropic's documentation gives no time. After the change, use the AI crawler check to confirm the rules are right, then run the same set of questions again a few weeks later.
Does Claude cite web pages every time?
No. Claude searches the web when it helps, and users can also ask it to search in their question. An answer without a search has no citations from that round, so when you test, make sure the answer shows sources.
Is there a way to guarantee Claude recommends me?
No. The answer differs every time and nobody can guarantee it. What you can do is make sure Claude's crawlers can read you and that you appear in the pages it cites, then watch the change as "how many times out of how many".
Sources
- Anthropic · Does Anthropic crawl data from the web, and how can site owners block the crawler? （Article dated 2026-04-07, checked 2026-11-04）
- Anthropic · Enabling and using web search （checked 2026-11-04）
- kagentx evaluations: the data on the case study pages; each example gives its date.