Can AI crawlers read a README on GitHub?
From public information, yes. github.com's robots.txt has a separate rule group for GPTBot, OAI-SearchBot, ClaudeBot, anthropic-ai and PerplexityBot, and it does not block repository home pages. But “can read” is not “will cite”: none of the three AI companies' official documentation says how they treat GitHub repository pages, and we could not verify whether ChatGPT actually cites READMEs.
| What we checked | Result (2026-11-06) | What it tells you |
|---|---|---|
| AI crawler group in github.com/robots.txt | One group for GPTBot, OAI-SearchBot, ClaudeBot, anthropic-ai and PerplexityBot, with a 1-second crawl delay; Bytespider is blocked from the whole site | These crawlers are named explicitly |
| Paths this group blocks | /*/raw/, /*/tree/, /*/archive/, /*/blame/, /*/commits/, /gist/, on-site search and more; repository home pages and /blob/ paths are not blocked | The README on a repository home page can be crawled; directory listings (/tree/) and raw files (/raw/) cannot |
| HTML of a repository home page | Requested with an OAI-SearchBot User-Agent string: HTTP 200; the README text is in the HTML; the title and meta description come from the repository name and description | No script rendering needed; this is our simulation, not what OpenAI's crawler actually fetched |
| raw.githubusercontent.com | No robots.txt (404); the raw README can be fetched | The raw URL is readable, but it is plain text with no repository name or description |
So the repository description (the one line in the About box) and the start of the README are the two pieces of text most likely to be read.
Where does a README live on GitHub, and what are the limits?
GitHub's documentation (About the repository README file, checked 2026-11-06) says a README can live in the hidden .github directory, the repository root or the docs directory, and if several exist the first in that order is shown; content beyond 500 KiB is truncated; links are relative to the current file, so relative paths also work in cloned repositories. For long documentation it recommends a wiki or a docs site.
How do I write a README that AI can cite?
The same as any page: answer “what is this and who is it for” first, then give details that can be checked. Here is where a README differs from an ordinary web page:
| Location | What goes in | Why |
|---|---|---|
| Repository description (About box) | One sentence: what it is, what language, what it does. No slogans | It goes into the page title and meta description (tested) |
| First paragraph of the README | Repeat and expand that sentence: name, what it is, supported platforms, one concrete number or limit | Read first and most likely to be lifted |
| Install and quick start | Copyable commands with versions and system requirements | “How do I install X” is the most common question |
| Comparison section | A table against neighbouring tools, each row with a source and a check date | Answers “how is X different from Y”; see the comparison pages guide |
| Links section | Full URLs for the website, docs and changelog | Sends AI to a page you fully control |
Write only checkable facts in the comparison section; see How to write comparison pages. For how to open the page so it can be lifted, see How to write content AI will quote: before and after.
What does a README template you can copy look like?
Below is a template (names and URLs are placeholders). Note that the first paragraph already says what it is, what language and what systems; the comparison table carries a check date; and the end lists full URLs for the website and docs.
# ExampleTool
ExampleTool is a command-line tool that converts Markdown notes into a static
site. It is written in Go and runs on macOS, Linux and Windows.
## Install
brew install exampletool
# or
go install github.com/example/exampletool@latest
## Quick start
exampletool build ./notes --out ./site
## ExampleTool vs. other static site generators
| | ExampleTool | Hugo | Jekyll |
|---|---|---|---|
| Language | Go | Go | Ruby |
| Input | Markdown notes | Markdown + templates | Markdown + Liquid |
| Config needed | none | config file | config file |
(Comparison checked 2026-11-06 against each project's own documentation.)
## Links
- Website: https://example.com/exampletool/
- Documentation: https://example.com/exampletool/docs/
- Changelog: https://github.com/example/exampletool/releases
If I only have a README, do I still need a website?
Yes. A README is limited by GitHub's page format: it cannot carry structured data or the things AI answers often quote, such as pricing and contact details, and the repository page has a lot of unrelated interface text. A website is the page you fully control. So: put the facts in the README and link to the site; repeat the same one-sentence definition on the site and add Does structured data help AI search?. For the site as a whole see How to get ChatGPT to recommend my product; for the “about” page see How to write an About page.
How do I fix it, step by step?
1. First make sure you are not blocking yourself
If your website's robots.txt blocks AI crawlers, the links in your README lead nowhere they can read. For the rules see How to write robots.txt for AI crawlers; for each crawler see AI crawler user agents: a side-by-side table.
2. Fill in the repository description and topics
Write a one-sentence description in the repository's About box and fill in the website URL. It appears in the page title and meta description.
3. Rewrite the first paragraph of the README
One-sentence definition plus one concrete fact (language, platform, version requirement). Do not start with badges and slogans.
4. Add install, quick start and a comparison table
Copyable commands; a check date on every comparison row.
5. Link the website and docs at the end
Use full URLs, not just “see the docs”.
6. Check again in a few weeks
Use the method in How do you measure AI visibility? Four metrics and a table you can follow: ask an AI “what tools can do X” and see whether it mentions you and which URL it cites.
FAQ
Does ChatGPT read my GitHub README?
We found no official documentation and no evidence of actual citations, so this is unverified. All we can confirm is that github.com's robots.txt (2026-11-06) does not block these AI crawlers from repository home pages.
Does GitHub let AI crawlers in?
The robots.txt we read on 2026-11-06 has one rule group for GPTBot, OAI-SearchBot, ClaudeBot, anthropic-ai and PerplexityBot with a 1-second crawl delay. It blocks paths such as raw, tree, archive, blame, commits, gist and on-site search, but not repository home pages. Bytespider is blocked from the whole site.
Do badges and images in a README help?
For AI, text inside images cannot be read. You can keep badges, but do not let them push out the first paragraph; put important information in text.
Is a longer README better?
No. GitHub truncates beyond 500 KiB, but the practical problem is that readers and crawlers look at the beginning first. Put the definition, install steps and comparison first, and move long documentation to a docs site.
What should the repository description say?
One sentence on what it is, what language it uses and what it does. It goes into the repository page's title and meta description (tested on one public repository on 2026-11-06), so it is the sentence most easily lifted.
Sources
- GitHub · github.com/robots.txt （Read 2026-11-06）
- GitHub Docs · About the repository README file （checked 2026-11-06）
- GitHub · 示例仓库页（用来查 HTML 里有什么） （Read 2026-11-06）
- kagentx evaluations: the data on the case study pages; each example gives its date.