Where does Grok find its answers?
According to xAI’s official documentation, from two places: posts on X, and web pages found by live search. The documentation (checked 2026-11-07; the page carries no date) lists two tools: X Search, which searches X posts, users and threads, and Web Search, which searches the web in real time and opens pages. The API automatically returns the source URLs for what it gathers. Which search index sits behind these tools, and how results are ranked, the documentation does not say.
| Question | What the official documentation says | What we do |
|---|---|---|
| Can Grok search content on X? | Stated: X Search searches posts, users and threads | Post concrete facts on X and link to your own site |
| Can Grok search web pages? | Stated: Web Search searches in real time and opens pages | The page must be readable on the spot, with the body text written directly in HTML |
| Do answers come with sources? | Stated: the API returns source URLs | Use the sources to find gaps |
| Which search index it uses | Not stated | Do not guess |
| How sources are chosen and ranked | Not stated | Ask three times yourself and read the sources |
| Crawler User-Agent | Not found | Not published by xAI; see the next section |
These documents describe xAI’s API tools. Whether Grok on the web and in the X app uses the same ones, the documentation does not say, and we have not verified it.
What is xAI's crawler called, and how do you allow it?
xAI has not published it, at least not in the documentation we read. We looked at the xAI docs overview page, the tools overview page and the Web Search page; none names a crawler or a User-Agent. Some third-party sites list a few names, but they disagree with each other and none cites xAI, so we have not verified them, and this guide does not put any of those names into robots.txt as fact.
This affects what you do in two ways:
- Without a name, you cannot allow or block Grok separately in robots.txt.
- When checking server logs, you also cannot identify it by User-Agent alone. See How to check your server logs for AI crawlers。
What you can do is avoid blocking it at a wider level: do not block the whole site with “User-agent: *” plus “Disallow: /”, and do not let a firewall challenge every unfamiliar bot. For how to write the rules see How to write robots.txt for AI crawlers; for the crawlers that have been published see AI crawler user agents: a side-by-side table.
What can an indie developer do to make Grok more likely to mention them?
The official documentation gives no ranking rules, so what follows is the basics, not a guarantee for Grok:
1. Make the web route work first
Web Search opens pages in real time, so the body text must be in the HTML and must not wait for scripts. Check it yourself first:
# Does your own site answer without scripts? (replace the URL) curl -s https://example.com/ | grep -c "your product name"
The reason is in Do AI crawlers render JavaScript?.
2. One page answers one question a buyer asks
Answer directly in the first two or three sentences, with concrete numbers, and give the source and date for data. For how to write it see How to write content AI will quote: before and after.
3. Have a clear account on X
X Search searches posts, users and threads. In your bio say what the product is and who it is for, and add your website URL; in posts state one concrete fact and put the full URL of the matching page on the last line.
4. Put what you want cited on your own site
Posts sink out of view and are not under your control. For what X’s robots.txt says to other AI crawlers see Do AI answers cite posts on X and LinkedIn?.
5. Answer other people’s questions honestly
When someone asks you a question on X, answer honestly and say you are the author; do not pose as a bystander.
How do you know whether Grok mentions you?
Ask yourself and read the sources. Grok’s answers differ each time, so start a new conversation and ask each question three times.
1. Write down 5 questions buyers ask
Leave out your brand name and write it the way a buyer would.
2. Start a new conversation for each question, ask three times, with web search on
Note three things: whether it mentions you, whom it recommends, and which URLs are in the sources (any x.com posts, and whether your own site appears).
3. Test again with the same questions two to four weeks later
For how to record it see How do you measure AI visibility? Four metrics and a table you can follow.
How is it different from other AI assistants?
Each assistant has different crawlers and sources, and each has its own guide: How to get ChatGPT to recommend my product, How to get Claude to recommend my product, How to get Perplexity to recommend my product, How to get Meta AI to recommend my product. What is distinctive about Grok is that the official documentation lists X posts as a separate search tool, while its crawler identity has not been published.
FAQ
Where do Grok’s answers come from?
In xAI’s official documentation Grok has two search tools: X Search searches X posts, users and threads, and Web Search searches the web in real time and opens pages. The API returns the source URL for each piece of information. The documentation does not say which search index is behind these tools.
What is xAI’s crawler User-Agent?
We did not find it in the official documentation, so we write “not published”. Third-party sites have listed a few names, but they contradict each other and none cites xAI; we have not verified them, and we do not recommend judging by User-Agent alone.
Can I allow Grok specifically in robots.txt?
With no officially published crawler name, there is no name to put in robots.txt. What we can confirm is that the default rules for ordinary web pages still apply to you, so do not block the whole site with “User-agent: *”.
Will posting on X get Grok to recommend me?
X Search searches posts on X, so that is one source the official documentation states. But the documentation does not say how results are ranked or which posts matter more, and we give no guarantee. Make your posts state concrete facts and link to the matching page on your own site.
Are answers in the Grok app the same as the API’s search tools?
The official documentation describes the API tools. Whether Grok on the web and in the X app uses the same set, the documentation does not say, and we have not verified it.
Sources
- xAI · Tools overview (Checked 2026-11-07; the page carries no date)
- xAI · Web Search (Checked 2026-11-07; the page carries no date)
- kagentx evaluations: the data on the case study pages; each example gives its date.