What do noindex, robots.txt Disallow and canonical each control?
The three control three different things: noindex controls whether a page appears in search results, robots.txt Disallow controls whether a crawler may fetch the page, and canonical controls which of several identical URLs counts. Google's official docs (viewed 2026-11-10) are clear on this; for the AI-crawler column the official docs say very little, and where they say nothing we mark it “not stated”.
| How it is written | What Google's docs say | What the AI crawlers' official docs say |
|---|---|---|
| noindex (meta robots or X-Robots-Tag) | It means “Do not show this page, media, or resource in search results”; the page must be crawlable for Google to see the rule (robots meta tag docs, page dated 2026-03-24) | Not stated: the crawler docs of OpenAI, Perplexity and Anthropic cover only robots.txt and do not mention noindex or X-Robots-Tag (each page read on 2026-11-10) |
| robots.txt Disallow | It controls crawling and is not a mechanism for keeping a page out of Google; a disallowed page can still be indexed if other sites link to it; and it stops Google from reading a noindex in the page (robots.txt intro, dated 2025-12-10) | This is the only control method their docs state: OpenAI says search-side changes take about 24 hours to take effect, and that ChatGPT-User is user-triggered so the rules “may not apply”; Perplexity-User generally does not follow it; Anthropic states that its three crawlers honor robots.txt |
| rel=canonical | “A strong signal”: a strong signal, not a command; within one site it advises against using noindex to pick the canonical page, and against using robots.txt for canonicalization (URL consolidation docs, dated 2026-07-10) | Not stated: none of the three docs mentions canonical |
How to read it: if you want a page out of Google results, use noindex and do not block that page in robots.txt. If you want a page not used by AI crawlers, the only control the official docs state is robots.txt; whether noindex does anything for them is unverified. For how to write robots.txt see How do you write robots.txt for AI crawlers?, and for canonical see Canonical tags and duplicate content: how to set them up on a multilingual site so pages are not treated as duplicates.
Which pages should be noindexed and which should not?
This is our own way of judging, not an official rule. There is one core question: is the page useful to users but without anything unique for search?
| Page type | Approach | Why |
|---|---|---|
| Useful entry pages with nothing unique (in our library, product pages without a full analysis) | noindex, follow, out of the sitemap, page kept | The page is useful to users on the site but no different from content elsewhere; putting it in search results only dilutes the site |
| Pages whose content is in another language (gammaslide.com's ar / fa / ru demo replay pages, which play recordings in other languages) | noindex, its own canonical, out of the sitemap | The page language does not match the URL language; people searching in that language cannot read it |
| On-site search results, sort and filter parameter pages | Disallow the parameters in robots.txt, or noindex | Nearly unlimited in number and duplicating the list page; the robots.txt of both notion.com and framer.com does this (see below) |
| Several URLs for the same content (with and without a slash, with and without parameters) | canonical, not noindex | Google's docs recommend canonical for choosing the canonical page, not noindex |
| Content that has been deleted | Return 404 or 410 | The content no longer exists; it is not thin content and does not need noindex |
| Account, checkout and admin pages | Require login, and disallow them in robots.txt | They are not content pages; crawling them is of no use |
For how to choose the status code for deleted content, see How do HTTP status codes affect AI crawlers? 404, 410, 301, 429, 5xx.
On 2026-11-10 we looked at the robots.txt of two large sites (the text only, not crawler behavior): notion.com disallows /templates/search?query=* for all crawlers; framer.com disallows, for all crawlers, URLs with parameters such as ?sort=, ?query=, ?tab= and ?language=. Both say, in effect, do not crawl search and filter results; the list and detail pages themselves are not blocked.
Which of our own pages are noindex?
Two places, both responses fetched directly with curl on 2026-11-10, verbatim:
$ curl -s https://kagentx.com/p/research-stammer | grep -i '<meta name="robots"' <meta name="robots" content="noindex, follow" /> $ curl -s https://kagentx.com/p/lindy | grep -i '<meta name="robots"' (no output: no robots meta, indexable by default) $ curl -s https://gammaslide.com/ar/demo/launch-plan | grep -i '<meta name="robots"' <meta name="robots" content="noindex" />
| URL | Result (2026-11-10) | Why |
|---|---|---|
| kagentx.com/p/research-stammer | HTTP 200, <meta name="robots" content="noindex, follow" />, canonical points to itself | A product page without a full analysis |
| kagentx.com/p/lindy | HTTP 200, no robots meta and no X-Robots-Tag response header | A product page with a full analysis, indexable |
| demo/launch-plan under gammaslide.com/ar/, /fa/ and /ru/ | HTTP 200, <meta name="robots" content="noindex" />, its own canonical; none of the /demo/ addresses for these three languages is in sitemap.xml | They play recordings in other languages and are deliberately not indexed |
| demo/launch-plan under gammaslide.com/ja/ and /zh-hans/ | HTTP 200, no robots meta | The content language matches the URL; indexable |
The kagentx library has 1,155 products in all. 141 of them have a full analysis (verdict, value chain, competition, risks), and those 141 pages may be indexed; that is also the number in the kagentx.com/agents/ list and in sitemap.xml (counted 2026-11-10, 141 each). The other 1,014 pages open normally but carry noindex and stay out of the sitemap. This design was set on 2026-10-21 and 10-28, and the rule is in the opening comment of the build script.
Whether ChatGPT, Perplexity or Claude have read or cited these noindex pages, we have no evidence and ran no experiment. All we know for sure is what the pages themselves return.
Search Console lists these pages as “excluded by noindex”; do you need to act?
No. Pages you marked noindex yourself showing up in the “not indexed” list is the expected result, not a fault. What to check is the opposite direction:
1. Export the list of URLs excluded by noindex
In Search Console's page indexing report, filter by this reason. The exact wording in the report is whatever your account shows; we did not check it item by item.
2. Compare it with your list of deliberate noindex pages
Every URL in the list should be on your list. Extra ones mean a page was marked by mistake.
3. Then look at the pages that should be indexed
Important pages should not appear under this reason. If one does, first check whether the template carries noindex to every page.
How do you add noindex to a batch of thin pages?
1. Sort the pages first
Using the table above, sort pages into: fill them out with content, noindex, merge with canonical, delete. Do not apply one rule to the whole site.
2. Put noindex in the server-rendered HTML
Put it in a meta robots tag in
, or send an X-Robots-Tag response header. Do not add it with front-end script: most AI crawlers do not run JavaScript, see Do AI crawlers render JavaScript?.3. Do not also block these pages in robots.txt
If you block them, Google cannot read the noindex. Let crawlers fetch the page and rely on noindex to keep it out of the index.
4. Take these pages out of the sitemap
Put only URLs you want indexed in the sitemap. For how to write sitemap dates, see How do you write a trustworthy sitemap lastmod?.
5. Verify with curl
Take one URL from each kind of page, confirm the response has noindex, and that robots.txt does not block it:
# Is the noindex in the HTML the crawler gets (not added by JavaScript)? curl -s https://example.com/some-page | grep -io '<meta name="robots"[^>]*>' # Or sent as a header? curl -sI https://example.com/some-page | grep -i x-robots-tag # The same URL must NOT be blocked in robots.txt, or crawlers never see the tag curl -s https://example.com/robots.txt | grep -i -B1 -A1 "some-page"
6. Review after two weeks
Check whether the number excluded in Search Console matches your list; you can also use AI crawler check to see whether robots.txt blocks AI crawlers by mistake. For how a large number of thin pages affects crawl volume, see Will AI crawlers crawl too much? Which crawlers honor Crawl-delay.
Another question about which pages others can read is covered in Can the GPT Store and AI agent directories get you recommended by AI?: a directory page being readable does not mean it is cited.
FAQ
Must thin pages be noindexed?
Not necessarily. First ask whether the page is useful to users and whether it can be filled out with unique content. If it can, fill it out; if it cannot but the page is still useful to users (such as a product entry or a replay page in another language), use noindex to keep it on the site but out of search results.
Will AI crawlers still read a noindex page?
The vendors do not say. The crawler docs of OpenAI, Perplexity and Anthropic cover only robots.txt and do not mention noindex, so whether each honors meta robots noindex is unverified. What is certain is that for Google, noindex is documented as “do not show in search results”.
Is robots.txt Disallow plus noindex a double safeguard?
No, it backfires. Google says that when a page is disallowed in robots.txt it cannot see the meta robots or X-Robots-Tag in the page, so the noindex is ignored; a disallowed page can still be indexed if other sites link to it. To keep a page out of the index, use only noindex and let crawlers fetch the page.
Search Console shows many pages excluded by noindex; is that a fault?
If you marked those pages noindex on purpose, it is the expected result, not a fault. What to watch for is the reverse: pages that should be indexed appearing under this reason, which means noindex was added to pages by mistake.
Should noindex pages be removed from the sitemap?
Yes. A sitemap tells crawlers “these pages should be indexed”, and listing noindex pages sends contradictory signals. kagentx keeps noindex pages out of the sitemap.
Sources
- Google · Robots meta tag, data-nosnippet, and X-Robots-Tag specifications (Viewed 2026-11-10; the page is dated 2026-03-24)
- Google · Introduction to robots.txt (Viewed 2026-11-10; the page is dated 2025-12-10)
- Google · What is URL canonicalization (Viewed 2026-11-10; the page is dated 2026-07-10)
- OpenAI · Overview of OpenAI Crawlers (checked 2026-11-10)
- Perplexity · Perplexity Crawlers (checked 2026-11-10)
- Anthropic · Does Anthropic crawl data from the web (checked 2026-11-10)
- kagentx · kagentx.com/p/research-stammer (Read 2026-11-10)
- kagentx · kagentx.com/p/lindy (Read 2026-11-10)
- kagentx evaluations: the data on the case study pages; each example gives its date.