How does Gemini relate to Google's search index?
Google's documentation treats the two products differently. AI Overviews and AI Mode: the docs say they use Google's search index, the page must be indexed and eligible to appear with a snippet, and there are no other requirements. The Gemini app: the “AI features and your website” page does not mention it; the Google-Extended page only says that supplying content from the Search index to the model at prompt time (grounding) is governed by Google-Extended.
So Google has not published how the Gemini app picks sources to recommend, and we do not guess. What is certain: being in Google's search index is a precondition for AI Overviews, and the Google documentation names Search index content as a source for Gemini grounding.
| Product | What Google's docs say | What you can do |
|---|---|---|
| AI Overviews / AI Mode | Uses the search index; the page must be indexed and eligible to appear with a snippet; no extra requirements | Do ordinary search well first; see the AI Overviews guide linked below |
| Gemini app | These two pages do not say how it picks sources; the Google-Extended page says grounding uses content from the Search index | Make sure Googlebot can crawl and the page is indexed; Google gives nothing more to act on |
| Gemini training | Google-Extended controls whether content can be used to train future Gemini models | To opt out of training, block Google-Extended |
What does Google-Extended control, and does blocking it drop me out of Search?
Google's common crawlers documentation (checked 2026-11-05) is specific: Google-Extended is a standalone robots.txt token that controls whether content Google crawls can be used to train future Gemini models, and also grounding in the Gemini app and Grounding with Google Search on Vertex AI. It does not affect whether a site is included in Google Search and is not a ranking signal.
It has no separate User-Agent. Crawling is still done by Google's existing crawlers; Google-Extended is only a control token you write in robots.txt, so you will not see visits named Google-Extended in server logs.
# Keep Google Search (and AI Overviews) open User-agent: Googlebot Allow: / # Opt out of Gemini training and grounding only User-agent: Google-Extended Disallow: /
Note: blocking Google-Extended also keeps your content out of Gemini grounding. If you want to be cited in Gemini, do not block it. To keep Search but opt out of training, the official method is also this token; there is no separate switch splitting training from grounding. This is our literal reading of the documentation; we found no Google statement on splitting them.
What are the steps?
1. First confirm Googlebot is not blocked
Enter your URL in the AI crawler check and see whether Googlebot is allowed; in Search Console's URL Inspection, check whether the page is indexed. AI Overviews and Gemini grounding both depend on the search index.
2. Decide whether to block Google-Extended
To be in Gemini training and grounding: do not block it. To opt out: add the Disallow from the snippet above. This is a business decision, not a ranking trick.
3. Keep pages eligible to appear with a snippet
Do not put nosnippet, max-snippet:0 or noindex on pages you want cited. Google's documentation lists them as ways to control display.
4. Answer one specific question per page, with body text in the HTML
Make the title the buyer's question and give the answer and numbers in the first paragraph. This is general practice; Google's documentation sets no separate requirement for Gemini. For the AI Overviews self-check, see How to appear in Google AI Overviews.
5. Ask Gemini your buyers' questions a few times
Ask each question 3 times in new chats and note whether you were mentioned and which URLs were cited. Google does not say how Gemini picks, so looking at results is the only way; what matters is how many of the three times you appear.
A side-by-side comparison of crawlers is in AI crawler user agents: a side-by-side table, and full robots.txt templates are in How to write robots.txt for AI crawlers.
FAQ
How do I get Google Gemini to recommend my product?
Google has not published how the Gemini app chooses. What you can do: let Googlebot crawl, keep pages indexed and eligible for a snippet, do not block Google-Extended, and ask Gemini your buyers' questions a few times to see results.
Will blocking Google-Extended drop me out of Google Search?
No. Google's documentation says it does not affect whether a site is included in Google Search and is not a ranking signal.
Why can't I see Google-Extended visits in my logs?
Google's documentation says it has no separate HTTP User-Agent; crawling uses Google's existing User-Agents, and it is only a control token in robots.txt.
If I block Google-Extended, will I still appear in AI Overviews?
AI Overviews use the search index, which depends on Googlebot's crawling and indexing, not on Google-Extended. Google's AI features documentation does not list Google-Extended as a condition for AI Overviews.
Does the Gemini app only cite top-ranked pages?
Google's documentation does not say, and we have no reliable data, so we draw no conclusion.
Sources
- Google Search Central · Google’s common crawlers （Checked 2026-11-05; the page carries no date）
- Google Search Central · AI features and your website （Checked 2026-11-05; the page carries no date）
- kagentx evaluations: the data on the case study pages; each example gives its date.