What do the five package registries say about AI crawlers?
Of the four registries whose robots.txt we could read, none blocks GPTBot, ClaudeBot or PerplexityBot by name, and package-page paths are not disallowed; the fifth, npm, put a Cloudflare challenge page in front of its robots.txt, so we could not read it. On 2026-11-09 we read each robots.txt with curl and then fetched one package page from each for the raw HTML.
| Registry | robots.txt (2026-11-09) | Are package-page paths disallowed? |
|---|---|---|
| npmjs.com | HTTP 403 with a Cloudflare challenge page (response header cf-mitigated: challenge) | Not verified: we could not read the rules |
| pypi.org | One group for *; /simple/, /packages/, /pypi/*/json, /search* and others are disallowed | /project/<name>/ is not disallowed |
| crates.io | One group for *, with an empty Disallow; the comment says allow crawling everything by default | Not disallowed |
| packagist.org | One group for *; /p/, /p2/ and /search/ are disallowed | /packages/<vendor>/<name> is not disallowed |
| rubygems.org | One group for *; /downloads/, /search?*, /names, /news and others are disallowed | /gems/<name> is not disallowed |
This is only what robots.txt says and what our own requests returned, not what an AI crawler really fetches. npm's challenge page means our programmatic request could not get the package page; whether real AI crawlers get the same result is not verified. For how to write crawler rules, see How do you write robots.txt for AI crawlers?.
Is the package page body in the raw HTML?
Of the four registries we could fetch, three write the summary straight into the HTML and one (crates.io) returns only an empty shell. We fetched one package page from each and checked whether the summary and README sentences appear in the body once scripts are removed.
| Package page | Format of title and meta description | Is the body in the HTML? |
|---|---|---|
| pypi.org/project/requests/ | title: requests · PyPI; the meta description is just the one-line Summary: Python HTTP for Humans. | In the HTML: HTTP 200, about 334 KB, with the README in the body |
| packagist.org/packages/monolog/monolog | title: monolog/monolog - Packagist.org; the meta description is the package description | In the HTML: the README is rendered in the body |
| rubygems.org/gems/rails | title: rails | RubyGems.org | your community gem host; there is no meta description tag | The summary sentence is in the HTML; the page has no README, only the summary, a version table and dependencies |
| crates.io/crates/serde | The title is always crates.io: Rust Package Registry, unrelated to the package | Not in the HTML: only a shell of about 5.4 KB; the summary and README load by script; a request without Accept: text/html even gets an empty 404 |
| registry.npmjs.org/express(npm 的 JSON 接口) | Not a web page | Returns description, keywords and readme (about 10,000 characters), but this is an interface for programs, not a package page |
This shows three things. First, the first thing at the top of a package page is the one-line description you wrote (PyPI also puts it in the meta description). Second, the README is the page body on PyPI and Packagist but not on RubyGems. Third, the crates.io page is rendered by script, so a crawler that does not run scripts cannot read the summary; see Do AI crawlers render JavaScript?. So do not leave “what the project is” only on the package page; your site and GitHub README need the same sentence.
What does an AI cite when answering “best npm package for X”?
We have no evidence. None of ChatGPT, Claude or Perplexity says which sources it reads when answering package questions, and we did not query a model to see actual citations, so whether it cites READMEs, blog posts or Stack Overflow is not verified. All we can say is about the input side: most of the package pages above can be read by crawlers, and PyPI and Packagist put the README in the body.
Without evidence, the safe approach is to say the same sentence everywhere a crawler might read: the package page description, the first paragraph of the README, the site home page and the docs home page. For how to write them, see {readme} and {docs}.
How should you write description, keywords and the first paragraph of the README?
Official docs (read on 2026-11-09): npm says description and keywords help people discover your package in npm search. PyPI shows the one-line Summary at the top of every project page and shows keywords as tags; classifiers are a fixed taxonomy maintained by PyPI, and PyPI does not check that you chose them accurately. The Python packaging specification says the Description may run to several paragraphs but should not contain the whole manual.
| Field | How to write it | Why |
|---|---|---|
| npm description / PyPI Summary | One sentence: what it is, what it does, and the required runtime and version (for example Node.js 18+). No slogans. | It appears at the top of the package page and in search results; PyPI also puts it in the meta description |
| keywords | 4 to 8 words users will search for: function, category and the usual names | The docs say it helps search; piling in unrelated words does no good |
| README 第一段 | Answer “what is this” first, then who it is for and a one-line install command; say the same thing as the description | It is the start of the body on the PyPI and Packagist package pages, and the start of the GitHub page |
| classifiers(PyPI) | Only pick the Python versions, systems and topics you really support | The docs say PyPI does not check accuracy; a wrong choice misleads users |
| homepage / project.urls | Fill in the full URLs of your site, docs and changelog | Sends people and programs that read the package page to a page you control |
{
"name": "notesite",
"version": "2.4.0",
"description": "Command-line tool that turns Markdown notes into a static site. Node.js 18+.",
"keywords": ["markdown", "static-site-generator", "cli", "notes"],
"homepage": "https://example.com/notesite/",
"repository": { "type": "git", "url": "git+https://github.com/example/notesite.git" }
}
[project] name = "notesite" description = "Command-line tool that turns Markdown notes into a static site. Python 3.9+." readme = "README.md" keywords = ["markdown", "static site generator", "cli", "notes"] classifiers = [ "Programming Language :: Python :: 3", "Topic :: Text Processing :: Markup :: Markdown", ] [project.urls] Homepage = "https://example.com/notesite/" Documentation = "https://example.com/notesite/docs/" Changelog = "https://github.com/example/notesite/releases"
For how to write the README overall, see Will AI search cite my GitHub README?; for more places beyond the README, see How do open source projects get recommended by AI search? Six places beyond the README.
Do deprecation marks and version numbers affect AI answers?
We have no evidence of the effect on AI answers, so it is not verified. What we can check is npm's own behavior: the official docs say deprecating a whole package removes it from search results on the npm website and shows a deprecation message on the package page. For example, for the latest version of the request package, 2.88.2, the deprecated field read on 2026-11-09 from registry.npmjs.org/request/latest is “request has been deprecated” plus a link to an explanation.
Two things follow for your package. First, if you no longer maintain it, say so with npm deprecate and point to a replacement, so old pages do not keep looking like a recommendation. Second, state the supported versions and the latest update date in the README and changelog; for how to write a changelog, see Do changelog pages help AI search?.
How do I fix it, step by step?
1. Look at your package page with your own request first
Fetch your package page with curl and confirm the description and the first paragraph of the README are in the raw HTML; if the npm page is blocked by a challenge page, look at the JSON at registry.npmjs.org/<package-name> and check the fields.
2. Rewrite the description and the first paragraph of the README
The same fact: what it is, what it does and which version it needs.
3. Fill in keywords, classifiers and links
Only real entries; fill in all three of homepage, docs and changelog URLs.
4. Put the same definition on your site
Use the same sentence on your site home page and docs home page, rendered in the HTML; see How should developer docs be written so AI can understand and cite them?.
5. If you stop maintaining it, deprecate it and point to a replacement
Use npm deprecate on npm, and say it at the top of the README on PyPI.
Other package marketplaces work in a similar way; for example, the page fields and README role of an Obsidian plugin are in How do I get my Obsidian plugin found in AI search?.
Command-line tools also have a Homebrew formula page and its JSON API; the test is in How do AI tools recommend command-line tools? Homebrew, GitHub and docs pages, tested.
FAQ
Can AI crawlers read npm and PyPI package pages?
On 2026-11-09 our curl request could fetch the PyPI package page but not the npm one (403, Cloudflare challenge page). What real AI crawlers get is not verified. PyPI's robots.txt names no AI crawler and does not disallow package pages.
What does an AI cite when it recommends npm packages?
There is no official statement and we have no evidence of actual citations, so it is not verified. That is why the package page, README and your site should all say the same sentence.
What should description and keywords say?
Write description as one sentence: what it is, what it does and the version it needs; write keywords as 4 to 8 words users search for. The official docs say they help people find the package in npm search.
Can an AI read the summary on a crates.io package page?
On 2026-11-09 the raw HTML we got was only a shell of about 5.4 KB, with no summary or README, and the title was unrelated to the package. A crawler that does not run scripts may not read it, so also put the summary in your GitHub README and on your site.
What happens to a deprecated package?
The npm docs say deprecating a whole package removes it from search results on the npm website and shows a deprecation message on the package page. The effect on AI answers is not verified.
Sources
- npm · package.json (checked 2026-11-09)
- npm · Deprecating and undeprecating packages or package versions (checked 2026-11-09)
- PyPI · Project Metadata (checked 2026-11-09)
- Python Packaging User Guide · Core metadata specifications (checked 2026-11-09)
- PyPI · pypi.org/robots.txt (Read 2026-11-09)
- crates.io · crates.io/robots.txt (Read 2026-11-09)
- Packagist · packagist.org/robots.txt (Read 2026-11-09)
- RubyGems · rubygems.org/robots.txt (Read 2026-11-09)
- npm · registry.npmjs.org/request/latest (Read 2026-11-09)
- kagentx evaluations: the data on the case study pages; each example gives its date.