Do API products' docs sites block AI crawlers?
Among the 12 well-known API products we checked, none blocks GPTBot, ClaudeBot or PerplexityBot by name. We read their robots.txt files on 2026-11-09: the main docs are Allow for all three crawlers, and what is blocked is mostly console, search, internal or parameter paths. Five of them put a Content-Signal line in robots.txt, and only Twilio sets ai-train=no.
| Product (docs site) | GPTBot | ClaudeBot | PerplexityBot | Notable in robots.txt |
|---|---|---|---|---|
| Stripe (docs.stripe.com) | Allow | Allow | Allow | Content-Signal: ai-train=yes, search=yes, ai-input=yes |
| OpenAI (developers.openai.com) | Allow | Allow | Allow | Allow: / |
| Anthropic (platform.claude.com) | Allow | Allow | Allow | Disallow: /api/ |
| Twilio (www.twilio.com/docs) | Allow | Allow | Allow | Content-Signal: ai-train=no, search=yes, ai-input=yes |
| Cloudflare (developers.cloudflare.com) | Allow | Allow | Allow | Content-Signal: ai-train=yes, search=yes, ai-input=yes |
| GitHub (docs.github.com) | Allow | Allow | Allow | User-agent: * ; no rules |
| Resend (resend.com/docs) | Allow | Allow | Allow | Content-Signal: ai-train=yes, search=yes, ai-input=yes |
| Supabase (supabase.com/docs) | Allow | Allow | Allow | Content-Signal: ai-train=yes, search=yes, ai-input=yes |
| Shopify (shopify.dev) | Allow (named block) | Allow (named block) | Allow (via *) | GPTBot, ClaudeBot, ChatGPT-User, Claude-User, CCBot named; same Disallow paths |
| Plaid (plaid.com/docs) | Allow | Allow | Allow | Disallow: /api/* /legacy-docs/ /sandbox/* *.pdf |
| Postman (www.postman.com) | Allow (named block) | Allow (named block) | Allow (named block) | Allow: / ; Disallow: /search?* ; llms.txt linked in comments |
| Mapbox (docs.mapbox.com) | Allow | Allow | Allow | Disallow: /archive/ |
How to read it: plain Allow or Allow (via *) means the crawler is not named and follows the User-agent: * rules; named block means it is named and has its own rules. The judgment covers docs paths only (for example /docs/…), is inferred from the text of robots.txt, and is not the behavior of real crawlers; a CDN or firewall may block separately, see How do you write robots.txt for AI crawlers?. For crawler names and purposes see AI crawler user agents: a side-by-side table.
Content-Signal is a line in robots.txt that states whether content may be used for model training (ai-train), search (search) and AI answer input (ai-input). Twilio allows search and AI answer input but not training; this is a stance it declares itself and does not change the Allow or Disallow result.
What should a pricing page say so AI treats you as a fitting API to recommend?
When users ask "which email API fits me", the answer compares price, usage tiers, rate limits and supported languages. All of these must be in the body text of the pricing page. For the general way to write a pricing page see How should a pricing page be written so AI answers correctly?; an API product needs the following in addition.
| What to write | How to write it | Why |
|---|---|---|
| Billing unit | Per thousand requests, per message, per million tokens, and so on; write the number and currency | AI needs one common unit to compare different APIs |
| Usage tier table | Monthly usage per tier, unit price and the price beyond the tier, as a table | A table is easier to read than an image or a slider |
| Rate limits | Requests per minute or per second, listed by tier | A hard criterion when developers choose an API |
| SLA and support | Availability commitment, support channels and response time, stating which tier has them | Answers "is it fit for production?" |
| Update date | Write on the page: "Pricing updated 2026-11" | The most common reason AI gets prices wrong is reading an old page |
Do not put prices only in a click-to-switch component or an image; fetch the page with curl and confirm the price numbers are in the returned HTML.
Should the OpenAPI spec be public?
We recommend making it public and findable from both the docs and the repository. All the products we checked put a public OpenAPI file on GitHub, and all of them opened on 2026-11-09. A spec is a machine-readable list of the interface: paths, parameters, response fields and authentication, which AI coding tools and search engines can read directly instead of guessing from doc pages. Whether AI answers read these files is not stated officially and we have not verified it.
| Product | Where the public spec lives | What we saw (2026-11-09) |
|---|---|---|
| Stripe | github.com/stripe/openapi | spec3.json opens |
| OpenAI | github.com/openai/openai-openapi | openapi.yaml, openapi: 3.1.0 |
| Twilio | github.com/twilio/twilio-oai | spec/json/twilio_api_v2010.json opens |
| GitHub | github.com/github/rest-api-description | api.github.com.json, openapi 3.0.3 |
| Plaid | github.com/plaid/plaid-openapi | 2020-09-14.yml, openapi 3.0.0 |
| Cloudflare | github.com/cloudflare/api-schemas | repository page opens |
| Resend | github.com/resend/resend-openapi | repository page opens |
We only confirmed that the addresses open; we did not check whether these specs are current or complete. What you can do yourself:
1. Put the spec at a fixed address
Keep one file in the repository, link to it from the docs site, and keep the address stable.
2. Fill in descriptions in the spec
One sentence per endpoint saying what it does; for each parameter give the unit and range of values.
3. Link to the spec from the docs
At the top of the "API reference" page write "OpenAPI spec: URL", and include it in the sitemap.
4. Update it with every release
The version number and date in the spec should match the changelog (see Do changelog pages help AI search?).
What should the status page and changelog contain?
A status page answers "is this API stable?" and a changelog answers "is it still maintained?". All 8 products we checked have a public status page, and all opened on 2026-11-09:
| Product | Status page |
|---|---|
| Stripe | status.stripe.com |
| OpenAI | status.openai.com |
| Anthropic | status.claude.com (also /api/v2/status.json, which returns a very short JSON) |
| Twilio | status.twilio.com |
| Cloudflare | www.cloudflarestatus.com |
| GitHub | www.githubstatus.com (also /api/v2/status.json) |
| Resend | resend-status.com |
| Supabase | status.supabase.com |
How: host the status page at status.your-domain; link to it from the site footer and the docs footer; link the SLA line on the pricing page to it; keep past incidents as readable text, not only charts. For changelogs see Do changelog pages help AI search?.
How should sample code be written so AI can cite it?
Give each endpoint one curl example that can be copied and run, then SDK examples in the main languages. Put the code in the HTML, not in images or in tabs that load only after a click.
curl https://api.example.com/v1/emails \
-H "Authorization: Bearer $EXAMPLE_API_KEY" \
-H "Content-Type: application/json" \
-d '{"from": "hello@example.com", "to": "you@example.org", "subject": "Hi", "text": "Hello"}'
1. Give both request and response
The full request command plus a returned JSON, with field names that match the spec.
2. State the install command and version
For example the npm or pip install command and the list of languages the SDK supports.
3. One task per page
Pages like "send an email with Node.js" or "how to verify a webhook signature", answering directly in the opening. See How to write use-case pages.
4. Write error codes as a table
Status code, meaning and how to handle it.
How do you get AI to mention you in "recommend an API" answers?
Besides the pages above, comparison questions usually cite comparison pages, review articles and developer community discussions.
1. First confirm your site does not block crawlers
Use AI crawler check to look at robots.txt, then check the CDN's bot rules.
2. Write 3 comparison pages
"Your API vs the two it is most often compared with", with a source and date for each item; see How to write comparison pages.
3. Write "best for whom" use-case pages
One scenario per page, such as "order notifications for online shops".
4. Add WebAPI structured data (optional)
Keep fields consistent with the body text; which AIs read it is not stated officially, see Does structured data help AI search?.
5. Ask AI again in a month
Ask "which API for scenario X" and see whether you are mentioned and which URLs are cited.
{
"@context": "https://schema.org",
"@type": "WebAPI",
"name": "Example Mail API",
"description": "REST API for sending transactional email; JSON over HTTPS.",
"documentation": "https://example.com/docs/",
"termsOfService": "https://example.com/terms/",
"provider": { "@type": "Organization", "name": "Example Inc.", "url": "https://example.com/" }
}
For how to write the docs site itself and whether to make llms.txt, see How do developer docs get cited by AI search? and What is llms.txt?; for the general recommendation questions of SaaS see How do SaaS products get recommended by ChatGPT?.
FAQ
Can AI crawlers read API docs sites?
Judging by robots.txt, yes: on 2026-11-09 we read 12 well-known API product docs sites and none blocks GPTBot, ClaudeBot or PerplexityBot by name. This is a judgment from the text of robots.txt; a CDN or firewall may block separately.
What is Content-Signal?
It is a line in robots.txt that says yes or no for ai-train (training), search (search) and ai-input (AI answer input). Stripe, Cloudflare, Resend and Supabase have yes for all three; Twilio has ai-train=no, search=yes, ai-input=yes (2026-11-09).
Will an OpenAPI spec make AI recommend my API?
We found no official statement and no actual evidence, so this is not verified. The benefit of a public spec is that it is machine-readable and AI coding tools can use it directly; the effect on "which API to recommend" is uncertain.
How should a pricing page be written so AI does not get it wrong?
Write the price, billing unit, usage tiers and update date as text and tables in the body, not only in images or click-to-switch components. Fetch the page with curl and confirm the price numbers are in the HTML.
Does a status page help AI recommendations?
There is no official statement, so this is not verified. At least it answers "is this API stable?"; all 8 products we checked have a public status page, and some offer a very short JSON endpoint.
Sources
- Stripe · docs.stripe.com/robots.txt (Read 2026-11-09)
- OpenAI · developers.openai.com/robots.txt (Read 2026-11-09)
- Anthropic · platform.claude.com/robots.txt (Read 2026-11-09)
- Twilio · www.twilio.com/robots.txt (Read 2026-11-09)
- Cloudflare · developers.cloudflare.com/robots.txt (Read 2026-11-09)
- Shopify · shopify.dev/robots.txt (Read 2026-11-09)
- Postman · www.postman.com/robots.txt (Read 2026-11-09)
- Stripe · stripe/openapi (Read 2026-11-09)
- GitHub · github/rest-api-description (Read 2026-11-09)
- GitHub · githubstatus.com status.json (Read 2026-11-09)
- kagentx evaluations: the data on the case study pages; each example gives its date.