Perplexity's two crawlers
Perplexity searches the web for every answer and marks its sources in the answer. Its documentation lists two crawlers that do different jobs:
| Crawler | What it does | Follows robots.txt? | Used for training? |
|---|---|---|---|
| PerplexityBot | Builds Perplexity's search index, so sites appear and are linked in its results | Yes | Not used for foundation model training |
| Perplexity-User | Opens pages live to answer a user's question | Generally not (the request comes from a user) | Not used for training |
Perplexity is closer to search results than ChatGPT: in Ahrefs' 2025-08 analysis, 28.6% of the URLs Perplexity cited ranked in Google's top 10, against 8% for ChatGPT. So on Perplexity, the classic work of writing good pages that rank pays off more.
Follow these 6 steps
1. Allow PerplexityBot
Enter your URL in the AI crawler check and see whether PerplexityBot is allowed. If not, add this to the top of your robots.txt:
User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: /
Perplexity says changes can take up to 24 hours. If a firewall or your CDN's bot blocking is on, you can allow the IP list Perplexity publishes (perplexity.ai/perplexitybot.json).
2. Put your text in the HTML itself
Perplexity cites the text it can read. Run this line with a sentence from the page inside the quotes; if it prints 0, switch to server-side rendering or prerendering:
curl -s https://your-site.com/pricing/ | grep -c "a sentence from the page"
3. One page, one specific question
Make the heading the buyer's question, answer it in the first paragraph, then give the steps, a comparison table, prices and limits. An answer that can be lifted out in a sentence or two is what others want to cite.
Give every number its source and date, show an updated date on the page, and fix outdated prices and features promptly.
4. Ask Perplexity your buyers' questions and read the Sources panel
Ask each question 3 times (a new thread each time) and write down the URLs in the Sources panel: whether you are there, whether competitors are, and what kind of page each source is.
- List articles and reviews: write to the author saying how you differ from the products on the list.
- Reddit and forum threads: answer from your own account and say you are the maker.
- Your own page is among the sources but you aren't recommended: rewrite that page's first paragraph to answer the question directly.
5. Add an llms.txt
Use the llms.txt generator to write a draft from your homepage and sitemap, edit it and put it in your site's root. It costs little, but it is not a rule every AI has committed to reading; the first four steps matter more.
6. Test again with the same questions a few weeks later
Perplexity's answers differ every time too. Compare how many answers include you, not where you ranked in one of them.
What kind of pages AI cites
Our evaluation of MochiStamp on 2026-10-22 (Japanese market): buyers' questions were put to AI (Claude Sonnet 5.5, web search on) 3 times, and 0 answers mentioned it. The AI read 10 cited pages; 10 listed products in the category and 0 mentioned it. The most cited were tutorials and blogs such as noripon.blog, ai-rise.net, note.com, not the tools' own sites.
That round asked Claude, not Perplexity, but the point is the same: to be cited, you need to appear on the kinds of pages AI finds, or write such pages yourself. Full data in the MochiStamp case study.
FAQ
Does Perplexity train models on my content?
Perplexity's documentation says PerplexityBot is for surfacing sites in search results and is not used to collect content for training foundation models; Perplexity-User is not used for training either.
I blocked Perplexity-User in robots.txt. Why do I still see its visits?
Perplexity's documentation says Perplexity-User fetches on a user's request and generally ignores robots.txt. To block it you have to use your firewall with the IP list it publishes; but then people who paste your link into Perplexity can't open it either.
If I rank well on Google, will Perplexity cite me?
Your chances are better, but it isn't certain. In Ahrefs' 2025-08 analysis, 28.6% of the URLs Perplexity cited ranked in Google's top 10, far more than ChatGPT's 8%, but most citations still came from outside the top 10.
How long until I'm cited?
There is no fixed time. Perplexity says robots.txt changes take up to 24 hours; whether a page is found and cited depends on the question and the competition. Test again with the same questions after 2–4 weeks.
Sources
- Perplexity · Perplexity Crawlers （checked 2026-11-03）
- Ahrefs · Only 12% of AI Cited URLs Rank in Google’s Top 10 for the Original Prompt （published 2025-08-11, 15,000 prompts）
- llmstxt.org · The /llms.txt file （proposed in 2024）
- kagentx evaluations: the data on the case study pages; each example gives its date.