Guide · Open source

How do open source projects get recommended by AI search? Six places beyond the README

Updated 2026-11-07 · kagentx

Short answer

Beyond the README, the text you can write includes package page descriptions and keywords (npm, PyPI), a docs site, the project website, awesome lists and release notes. On 2026-11-07 we read pypi.org’s robots.txt: it has no separate rules for AI crawlers and does not block project pages. npm’s robots.txt was stopped by a Cloudflare challenge page, so we did not read it and its rules are unverified. Nobody documents whether AI cites package pages and we have no evidence either, so write the project’s definition on your own website and docs site as well, and treat package descriptions and keywords as input for npm’s and PyPI’s own search.

Beyond the README, where else can AI learn about an open source project?

There are six places where you can write text you control: package pages (npm, PyPI), a docs site, the project website, awesome lists, release notes, and the README itself. The README has its own guide: Will AI search cite my GitHub README?. The table below covers the other five. None of the three AI companies documents which of these places its product reads, so we have not verified that; the table only contains what we could check.

LocationWhat you can writeWhat we checked
npm / PyPI package pageA one-line description, keywords, homepage and docs links, the READMEThe purpose of each field is documented; both sites showed a challenge page to scripted requests (next section)
Docs siteOne question per page, version numbers, update datesSee the developer docs guide; our llms.txt sample of 11 docs sites is in that one
Project websiteA one-sentence definition, a comparison table, install steps, pricing or licenseThe only page you fully control; you can add structured data
Awesome listsBeing included in someone else’s curated list, with a one-line descriptionEach list has its own rules; sindresorhus/awesome had paused pull requests on 2026-11-07
Release notesWhat changed in each version and which versions it requiresGitHub releases are based on Git tags; notes can be written by hand or generated

What is the attitude of npm and PyPI toward AI crawlers?

On 2026-11-07 we requested both robots.txt files. PyPI has no separate rules for AI crawlers and does not block project pages. We could not read npm’s robots.txt because the request was stopped by a Cloudflare challenge page. Neither result proves whether an AI crawler can actually fetch a package page, or that it would cite one.

What we didResult (2026-11-07)What it tells you
Read pypi.org/robots.txtHTTP 200. One “User-agent: *” group plus a Sitemap line; it blocks /simple/, /packages/, /pypi/*/json, /search*, /account/ and a few others; no separate rules for GPTBot, ClaudeBot or PerplexityBotRead literally, AI crawlers are treated like any other crawler: project pages at /project/<name>/ are allowed, package files and the JSON API are not
Read www.npmjs.com/robots.txtHTTP 403 with a Cloudflare “Just a moment...” challenge page; the same 403 with the User-Agent strings of GPTBot and ClaudeBot; WebFetch also got 403We did not get the rules, so they are unverified. It shows npm challenges scripted requests like ours; whether real AI crawlers meet the same challenge, we do not know
Open pypi.org/project/requests/ (WebFetch)We got a “Client Challenge” page, with no summary, README or sidebar linksPackage pages can also show a challenge to scripted requests; whether AI crawlers can actually fetch them is unverified

So do not put “what the project is” only on the package page. Descriptions and keywords mainly serve npm’s and PyPI’s own search; to be read by AI, write the same definition on your own website and docs site. See How do you write robots.txt for AI crawlers? and AI crawler user agents: a side-by-side table.

How should I write the npm description and keywords?

The official docs (npm Docs, package.json, checked 2026-11-07) say that description and keywords both help people find your package in npm search; homepage is the project homepage, repository is where the code lives, and README and LICENSE are always included in the package. npm search matches the query against a package’s title, description, readme and keywords and orders results by match, with a small penalty for spammy or brand-new packages; a newly published package can take up to two weeks to show up, and deprecated packages are left out.

FieldHow to write itWhy
descriptionOne sentence: what it is, what it does, the supported runtime and version. No slogansIt takes part in npm search matching and is the first line shown on the package page
keywords4 to 8 terms people search for: the function, the category, what competitors call itThe docs say it also helps discovery; stuffing in unrelated words makes a package look spammy and can be down-ranked
homepageThe full URL of your website or docs home pageSends people and programs that read the package page to a page you control
repositoryThe repository addressLets people and the npm repo command find the code
{
  "name": "examplemd",
  "version": "2.4.0",
  "description": "Command-line tool that converts Markdown notes into a static site. Node.js 18+.",
  "keywords": ["markdown", "static-site-generator", "cli", "notes"],
  "homepage": "https://example.com/examplemd/",
  "repository": { "type": "git", "url": "git+https://github.com/example/examplemd.git" }
}

How should I write the PyPI description, keywords and project links?

The Python Packaging User Guide (Writing your pyproject.toml, checked 2026-11-07) says: description is a one-line summary shown as the headline of the project page and in search result lists; readme is the long description on the project page; keywords help PyPI’s search box suggest your project; classifiers are used only for searching and browsing; urls appear in the left sidebar of the project page, and well-known labels such as Homepage, Documentation and Changelog have specific meanings.

FieldHow to write itWhy
descriptionA one-line summary: what it is and the supported Python versionsShown as the headline of the project page and in search results
readmePoint it to README.md and write the opening as the README guide describesIt is the long description on the project page, so the PyPI page and the GitHub page share the same text
keywordsTerms people search forThe docs say it helps the search box suggest your project
project.urlsAt least fill in the three well-known labels Homepage, Documentation and ChangelogShown in the left sidebar of the project page; the well-known labels have specific meanings
[project]
name = "examplemd"
description = "Command-line tool that converts Markdown notes into a static site. Python 3.9+."
readme = "README.md"
keywords = ["markdown", "static site generator", "cli", "notes"]

[project.urls]
Homepage = "https://example.com/examplemd/"
Documentation = "https://example.com/examplemd/docs/"
Changelog = "https://github.com/example/examplemd/releases"

What are awesome lists and release notes good for?

An awesome list is a curated list maintained by someone else. Being included means a one-line linked description on a page someone else wrote, the same kind of source as a “best tools” article; see How do I get into “best X tools” list articles?. Each list has its own rules, so read its contribution guide before submitting. For sindresorhus/awesome itself, on 2026-11-07 its page said pull requests were temporarily disabled; the details are in the contributing.md it links to, which we did not read.

Release notes are the text an AI is most likely to read when answering “what changed in this version” or “what should I watch out for when upgrading”. GitHub releases are based on Git tags, and the notes can be written by hand or generated from a default or custom template (GitHub Docs, About releases, checked 2026-11-07). The docs do not say what release notes should contain; the layout below is our own suggestion: the first paragraph states the most important change and the version requirements, changes are grouped, and breaking changes come with a migration path.

## 2.4.0 (2026-11-07)

Adds incremental builds: unchanged notes are skipped, so a rebuild of 2,000
notes takes seconds instead of minutes. Requires Node.js 18 or later.

### Changed
- `--out` now defaults to ./site (was ./dist). Migration: pass --out ./dist to keep the old path.

### Fixed
- Wiki links with spaces in the target no longer break the build (#312).

For how to write an opening that can be lifted out, see How to write content AI will quote: before and after; for what structured data to add to the website, see Does structured data help AI search?.

If your project is an MCP server or a model on Hugging Face, each has its own directory page and metadata: see How do you get an MCP server recommended by AI and listed in connector directories? and How do models and Spaces on Hugging Face get recommended by AI?.

How do I fix it, step by step?

  1. 1. Fix the README first

    The README is the body text of the GitHub page, the npm page and the PyPI page. Rewrite its first paragraph following Will AI search cite my GitHub README? and all three benefit.

  2. 2. Update the package description and keywords

    One sentence saying what it is, what it does and which versions it needs; keywords only for terms people search. For npm this goes in package.json, for PyPI in pyproject.toml.

  3. 3. Fill in the homepage, docs and changelog links

    Fill in homepage on npm, and Homepage, Documentation and Changelog on PyPI. These are the exits that lead people and crawlers to your own site.

  4. 4. Use the same definition on the website and docs site

    Define the project in one sentence at the top of the website and give the docs site one question per page; see How should developer docs be written so AI can understand and cite them?. Check that your own robots.txt does not block AI crawlers.

  5. 5. Write proper release notes for every release

    State the most important change and the version requirements in the first paragraph, give a migration path for breaking changes, and put the date in the heading.

  6. 6. Submit to a few relevant awesome lists

    Read each list’s contribution guide first, write a one-line description in its format, and say plainly what the project is without exaggerating.

  7. 7. Test again with the same questions a few weeks later

    Ask AI “which open source tools can do X”, three times per question, and note how often you appear and which URLs are cited; the method is in How do you measure AI visibility? Four metrics and a table you can follow. Remember that a newly published npm package can take up to two weeks to appear in npm search.

If your project is a WordPress plugin, see How do WordPress plugins get recommended by AI? for how to write the directory page and readme.txt.

If your project is a Linux desktop app, see How do Linux apps get recommended by AI search? for how to read Flathub, Snap Store and AUR pages.

If the project ships as a Docker image or a GitHub Action, see How do AI tools recommend Docker images and GitHub Actions? Docker Hub and Marketplace pages, tested for a test of the Docker Hub and Marketplace pages.

If you build a Blender add-on, see How do AI search tools find Blender add-ons? The Extensions platform tested, plus the official listing requirements for the Extensions platform's robots.txt, the detail page HTML test and the official listing requirements.

FAQ

Do npm keywords help in AI search?

The official docs only say keywords make a package easier to find in npm search. Whether AI search reads keywords is not documented, and we have not verified it.

Does PyPI allow AI crawlers?

The pypi.org/robots.txt we read on 2026-11-07 has a single “User-agent: *” group and no separate rules for GPTBot, ClaudeBot, PerplexityBot and the like. It blocks paths such as /simple/, /packages/, /pypi/*/json, /search* and /account/; project pages are not blocked.

What does npm’s robots.txt say about AI crawlers?

Unverified. On 2026-11-07 we requested www.npmjs.com/robots.txt and got a Cloudflare challenge page (HTTP 403), not the file.

How long until a newly published package shows up in npm search?

The npm docs say a newly published package can take up to two weeks to appear in search, and deprecated packages do not appear.

Does an open source project need a website?

We recommend one. Package pages and READMEs are limited by the platform’s format, cannot hold structured data, and are surrounded by unrelated interface text; a website is the only page you fully control. One page is enough: a one-sentence definition, install steps, a comparison table and a link to the docs.

Sources

kagentx

Can crawlers read your pages?
Let the evaluation tell you

The evaluation checks whether crawlers can get in and whether the text is served directly, then asks ChatGPT, Perplexity and Claude (with web search on) your buyers' questions and sees whether you are mentioned. The Basic plan is $1, and results usually take 2–3 minutes.