What is llms.txt?
llms.txt is a Markdown file placed at the root of a site (your-site.com/llms.txt) that gives AI a curated summary of the site and a list of links: who you are, which pages matter, and what each covers. Jeremy Howard proposed it on 2024-09-03, and the specification is at llmstxt.org. As proposed, an AI reads the file while answering a question and then follows the links to the detail.
It is a community proposal, not an internet standard. robots.txt has a formal standard (RFC 9309) and llms.txt does not, so whether and how each AI reads it is up to that AI.
What does an llms.txt file look like?
Under the llmstxt.org specification the file is Markdown with the parts below in order; only the first is required:
- An H1 with the name of the project or site (the only required part).
- A blockquote summarizing the site in a sentence or two.
- Zero or more paragraphs of extra detail, in any Markdown other than headings.
- Zero or more H2 sections, each a list of links: [name](url), optionally followed by a colon and a note.
- An H2 section named Optional for secondary links that the AI can skip when it needs less context.
The proposal also suggests offering a clean Markdown version of each page, at the original URL with .md added. Here is an example written to the specification:
# Acme Invoices > Acme Invoices is invoicing software for solo consultants. Plans start at $9 a month; it supports PDF export and 12 currencies. Acme works with Stripe and PayPal. It does not offer payroll. ## Product - [Features](https://acme.example/features.md): what the app can do - [Pricing](https://acme.example/pricing.md): plans, limits and refund policy ## Docs - [Quick start](https://acme.example/docs/start.md): create and send your first invoice - [API reference](https://acme.example/docs/api.md): REST endpoints and authentication ## Optional - [Changelog](https://acme.example/changelog.md): release notes
Does llms.txt actually work?
The current state: Google has said publicly that it does not use it, and the AI companies have not officially said their crawlers read it. Below are the statements with a traceable source, each checked on 2026-11-04:
| Who | says | Source and date |
|---|---|---|
| Google (John Mueller) | Wrote on Bluesky that no AI system currently uses llms.txt. He also said consumer chatbots fetch your pages but none of them fetch llms.txt. | Bluesky post of 2025-06-17, reported by Search Engine Roundtable on 2025-06-18 |
| Google (Search Central documentation) | Its guide to generative AI features states that you do not need to create llms.txt or similar files to appear in Google Search, that Google Search itself does not use them and that they will not affect your visibility; making them for other services is fine. | The guide page on developers.google.com, which carries no date; viewed 2026-11-04 |
| OpenAI | No official confirmation. The crawler documentation (the bots page) does not say GPTBot, OAI-SearchBot or ChatGPT-User read other sites' llms.txt. | Viewed 2026-11-04 |
| Anthropic | No official confirmation. The crawler documentation does not say ClaudeBot, Claude-SearchBot or Claude-User read llms.txt. | Viewed 2026-11-04 |
| Perplexity | No official confirmation. The crawler documentation does not say PerplexityBot or Perplexity-User read llms.txt. | Viewed 2026-11-04 |
One easily misread fact: the developer documentation sites of OpenAI, Perplexity and Anthropic all publish an llms.txt of their own (all opened fine on 2026-11-04). That shows they are happy for other people's AI tools to read their docs; it does not mean their search crawlers read your llms.txt.
So there is currently no public evidence that having an llms.txt gets you recommended more in ChatGPT, Perplexity, Claude or Google. Do not expect it to earn you recommendations.
When is llms.txt worth doing?
| Situation | Worth it? | Cause |
|---|---|---|
| You have developer or API docs | Yes | The proposal's aim is to help AI tools understand a site; it costs very little for a docs site, and the docs sites of OpenAI, Anthropic and Perplexity publish one too. |
| You run an ordinary product or content site and want AI to recommend it | You can, but later in the order | There is no evidence it brings recommendations. First allow the crawlers, put the text in the HTML and get into the pages AI cites; then do this. |
| You hope it will raise your Google ranking | No | Google's documentation states that Google Search ignores it. |
The cost is mainly upkeep: when a price or feature changes, the file has to change too, or you are giving AI an out-of-date summary. Google's documentation says it will not harm your search performance.
How do I make an llms.txt?
1. Get the home and pricing pages right first
llms.txt is only a list of links, and what the AI sees when it follows one is still your page. The body text must be in the HTML; see Do AI crawlers render JavaScript.
2. Draft the file
Use the llms.txt generator: enter your URL and it reads the name, summary and main pages from your home page and sitemap and produces a draft laid out to the specification.
3. Rewrite it in your own words
Write the product name in the H1 and a sentence or two in the blockquote: what you are, who it is for, what it costs. Put only the most important pages in the link lists (features, pricing, docs, about) and move secondary ones to Optional. Give numbers a source and a date.
4. Put it in the root and check it opens
The URL is your-site.com/llms.txt and it should return 200 as plain text or Markdown. If you use a firewall or CDN, also check with the AI crawler check that crawlers can get in.
5. See whether anyone reads it
After a few weeks, search your server logs for requests to /llms.txt and see which User-Agents made them. A log entry is the only sign a crawler reads it; with none, don't expect anything.
Things to do before llms.txt: How to write robots.txt for AI crawlers, and AI visibility audit checklist.
FAQ
Does llms.txt improve recommendations or ranking in ChatGPT or Google?
There is no public evidence. Google's documentation says Google Search ignores it, and OpenAI, Anthropic and Perplexity have not said their crawlers read it. It does no harm to search performance, but don't expect an effect.
Can llms.txt replace robots.txt?
No, they are two different things. robots.txt decides whether a crawler may fetch your pages and has a formal standard; llms.txt is a summary and list of links for AI, only a proposal, and controls no access.
Do I have to follow the llmstxt.org format?
To be called llms.txt, follow its specification: Markdown, where the H1 is the only required part and the rest is optional. Beyond that format, nobody guarantees that any given AI will read it.
How do I know whether an AI has read my llms.txt?
Search your server or CDN logs for requests to /llms.txt and look at the User-Agent. If there are no requests, then at least no crawler has come to read it.
What should go in llms.txt?
Put in the most important pages: product features, pricing, docs and about, each with a one-line note, and numbers with a source and date. Put secondary material in the Optional section. Don't stuff it with every link on the site.
Sources
- llmstxt.org · The /llms.txt file （proposed in 2024）
- Search Engine Roundtable · Google Says No AI System Currently Uses LLMs.txt （Published 2025-06-18, checked 2026-11-04; quotes John Mueller's Bluesky post of 2025-06-17）
- Google · Google’s Guide to Optimizing for Generative AI Features on Google Search （Viewed 2026-11-04; the page carries no date）
- OpenAI · Overview of OpenAI Crawlers （checked 2026-11-04）
- Perplexity · Perplexity Crawlers （checked 2026-11-04）
- Anthropic · Does Anthropic crawl data from the web, and how can site owners block the crawler? （checked 2026-11-04）
- kagentx evaluations: the data on the case study pages; each example gives its date.