Beyond the README, where else can AI learn about an open source project?
There are six places where you can write text you control: package pages (npm, PyPI), a docs site, the project website, awesome lists, release notes, and the README itself. The README has its own guide: Will AI search cite my GitHub README?. The table below covers the other five. None of the three AI companies documents which of these places its product reads, so we have not verified that; the table only contains what we could check.
| Location | What you can write | What we checked |
|---|---|---|
| npm / PyPI package page | A one-line description, keywords, homepage and docs links, the README | The purpose of each field is documented; both sites showed a challenge page to scripted requests (next section) |
| Docs site | One question per page, version numbers, update dates | See the developer docs guide; our llms.txt sample of 11 docs sites is in that one |
| Project website | A one-sentence definition, a comparison table, install steps, pricing or license | The only page you fully control; you can add structured data |
| Awesome lists | Being included in someone else’s curated list, with a one-line description | Each list has its own rules; sindresorhus/awesome had paused pull requests on 2026-11-07 |
| Release notes | What changed in each version and which versions it requires | GitHub releases are based on Git tags; notes can be written by hand or generated |
What is the attitude of npm and PyPI toward AI crawlers?
On 2026-11-07 we requested both robots.txt files. PyPI has no separate rules for AI crawlers and does not block project pages. We could not read npm’s robots.txt because the request was stopped by a Cloudflare challenge page. Neither result proves whether an AI crawler can actually fetch a package page, or that it would cite one.
| What we did | Result (2026-11-07) | What it tells you |
|---|---|---|
| Read pypi.org/robots.txt | HTTP 200. One “User-agent: *” group plus a Sitemap line; it blocks /simple/, /packages/, /pypi/*/json, /search*, /account/ and a few others; no separate rules for GPTBot, ClaudeBot or PerplexityBot | Read literally, AI crawlers are treated like any other crawler: project pages at /project/<name>/ are allowed, package files and the JSON API are not |
| Read www.npmjs.com/robots.txt | HTTP 403 with a Cloudflare “Just a moment...” challenge page; the same 403 with the User-Agent strings of GPTBot and ClaudeBot; WebFetch also got 403 | We did not get the rules, so they are unverified. It shows npm challenges scripted requests like ours; whether real AI crawlers meet the same challenge, we do not know |
| Open pypi.org/project/requests/ (WebFetch) | We got a “Client Challenge” page, with no summary, README or sidebar links | Package pages can also show a challenge to scripted requests; whether AI crawlers can actually fetch them is unverified |
So do not put “what the project is” only on the package page. Descriptions and keywords mainly serve npm’s and PyPI’s own search; to be read by AI, write the same definition on your own website and docs site. See How do you write robots.txt for AI crawlers? and AI crawler user agents: a side-by-side table.
How should I write the npm description and keywords?
The official docs (npm Docs, package.json, checked 2026-11-07) say that description and keywords both help people find your package in npm search; homepage is the project homepage, repository is where the code lives, and README and LICENSE are always included in the package. npm search matches the query against a package’s title, description, readme and keywords and orders results by match, with a small penalty for spammy or brand-new packages; a newly published package can take up to two weeks to show up, and deprecated packages are left out.
| Field | How to write it | Why |
|---|---|---|
| description | One sentence: what it is, what it does, the supported runtime and version. No slogans | It takes part in npm search matching and is the first line shown on the package page |
| keywords | 4 to 8 terms people search for: the function, the category, what competitors call it | The docs say it also helps discovery; stuffing in unrelated words makes a package look spammy and can be down-ranked |
| homepage | The full URL of your website or docs home page | Sends people and programs that read the package page to a page you control |
| repository | The repository address | Lets people and the npm repo command find the code |
{
"name": "examplemd",
"version": "2.4.0",
"description": "Command-line tool that converts Markdown notes into a static site. Node.js 18+.",
"keywords": ["markdown", "static-site-generator", "cli", "notes"],
"homepage": "https://example.com/examplemd/",
"repository": { "type": "git", "url": "git+https://github.com/example/examplemd.git" }
}
How should I write the PyPI description, keywords and project links?
The Python Packaging User Guide (Writing your pyproject.toml, checked 2026-11-07) says: description is a one-line summary shown as the headline of the project page and in search result lists; readme is the long description on the project page; keywords help PyPI’s search box suggest your project; classifiers are used only for searching and browsing; urls appear in the left sidebar of the project page, and well-known labels such as Homepage, Documentation and Changelog have specific meanings.
| Field | How to write it | Why |
|---|---|---|
| description | A one-line summary: what it is and the supported Python versions | Shown as the headline of the project page and in search results |
| readme | Point it to README.md and write the opening as the README guide describes | It is the long description on the project page, so the PyPI page and the GitHub page share the same text |
| keywords | Terms people search for | The docs say it helps the search box suggest your project |
| project.urls | At least fill in the three well-known labels Homepage, Documentation and Changelog | Shown in the left sidebar of the project page; the well-known labels have specific meanings |
[project] name = "examplemd" description = "Command-line tool that converts Markdown notes into a static site. Python 3.9+." readme = "README.md" keywords = ["markdown", "static site generator", "cli", "notes"] [project.urls] Homepage = "https://example.com/examplemd/" Documentation = "https://example.com/examplemd/docs/" Changelog = "https://github.com/example/examplemd/releases"
What are awesome lists and release notes good for?
An awesome list is a curated list maintained by someone else. Being included means a one-line linked description on a page someone else wrote, the same kind of source as a “best tools” article; see How do I get into “best X tools” list articles?. Each list has its own rules, so read its contribution guide before submitting. For sindresorhus/awesome itself, on 2026-11-07 its page said pull requests were temporarily disabled; the details are in the contributing.md it links to, which we did not read.
Release notes are the text an AI is most likely to read when answering “what changed in this version” or “what should I watch out for when upgrading”. GitHub releases are based on Git tags, and the notes can be written by hand or generated from a default or custom template (GitHub Docs, About releases, checked 2026-11-07). The docs do not say what release notes should contain; the layout below is our own suggestion: the first paragraph states the most important change and the version requirements, changes are grouped, and breaking changes come with a migration path.
## 2.4.0 (2026-11-07) Adds incremental builds: unchanged notes are skipped, so a rebuild of 2,000 notes takes seconds instead of minutes. Requires Node.js 18 or later. ### Changed - `--out` now defaults to ./site (was ./dist). Migration: pass --out ./dist to keep the old path. ### Fixed - Wiki links with spaces in the target no longer break the build (#312).
For how to write an opening that can be lifted out, see How to write content AI will quote: before and after; for what structured data to add to the website, see Does structured data help AI search?.
If your project is an MCP server or a model on Hugging Face, each has its own directory page and metadata: see How do you get an MCP server recommended by AI and listed in connector directories? and How do models and Spaces on Hugging Face get recommended by AI?.
How do I fix it, step by step?
1. Fix the README first
The README is the body text of the GitHub page, the npm page and the PyPI page. Rewrite its first paragraph following Will AI search cite my GitHub README? and all three benefit.
2. Update the package description and keywords
One sentence saying what it is, what it does and which versions it needs; keywords only for terms people search. For npm this goes in package.json, for PyPI in pyproject.toml.
3. Fill in the homepage, docs and changelog links
Fill in homepage on npm, and Homepage, Documentation and Changelog on PyPI. These are the exits that lead people and crawlers to your own site.
4. Use the same definition on the website and docs site
Define the project in one sentence at the top of the website and give the docs site one question per page; see How should developer docs be written so AI can understand and cite them?. Check that your own robots.txt does not block AI crawlers.
5. Write proper release notes for every release
State the most important change and the version requirements in the first paragraph, give a migration path for breaking changes, and put the date in the heading.
6. Submit to a few relevant awesome lists
Read each list’s contribution guide first, write a one-line description in its format, and say plainly what the project is without exaggerating.
7. Test again with the same questions a few weeks later
Ask AI “which open source tools can do X”, three times per question, and note how often you appear and which URLs are cited; the method is in How do you measure AI visibility? Four metrics and a table you can follow. Remember that a newly published npm package can take up to two weeks to appear in npm search.
If your project is a WordPress plugin, see How do WordPress plugins get recommended by AI? for how to write the directory page and readme.txt.
If your project is a Linux desktop app, see How do Linux apps get recommended by AI search? for how to read Flathub, Snap Store and AUR pages.
If the project ships as a Docker image or a GitHub Action, see How do AI tools recommend Docker images and GitHub Actions? Docker Hub and Marketplace pages, tested for a test of the Docker Hub and Marketplace pages.
If you build a Blender add-on, see How do AI search tools find Blender add-ons? The Extensions platform tested, plus the official listing requirements for the Extensions platform's robots.txt, the detail page HTML test and the official listing requirements.
FAQ
Do npm keywords help in AI search?
The official docs only say keywords make a package easier to find in npm search. Whether AI search reads keywords is not documented, and we have not verified it.
Does PyPI allow AI crawlers?
The pypi.org/robots.txt we read on 2026-11-07 has a single “User-agent: *” group and no separate rules for GPTBot, ClaudeBot, PerplexityBot and the like. It blocks paths such as /simple/, /packages/, /pypi/*/json, /search* and /account/; project pages are not blocked.
What does npm’s robots.txt say about AI crawlers?
Unverified. On 2026-11-07 we requested www.npmjs.com/robots.txt and got a Cloudflare challenge page (HTTP 403), not the file.
How long until a newly published package shows up in npm search?
The npm docs say a newly published package can take up to two weeks to appear in search, and deprecated packages do not appear.
Does an open source project need a website?
We recommend one. Package pages and READMEs are limited by the platform’s format, cannot hold structured data, and are surrounded by unrelated interface text; a website is the only page you fully control. One page is enough: a one-sentence definition, install steps, a comparison table and a link to the docs.
Sources
- PyPI · pypi.org/robots.txt (Read 2026-11-07)
- npm · www.npmjs.com/robots.txt (returned a challenge page; file not read) (tried to read on 2026-11-07)
- npm Docs · package.json (checked 2026-11-07)
- npm Docs · Searching for and choosing packages to download (checked 2026-11-07)
- Python Packaging User Guide · Writing your pyproject.toml (checked 2026-11-07)
- GitHub Docs · About releases (checked 2026-11-07)
- GitHub · sindresorhus/awesome (Read 2026-11-07)
- kagentx evaluations: the data on the case study pages; each example gives its date.