Guide · API products

How do you get an API product recommended by AI?

Updated 2026-11-09 · kagentx

Short answer

First confirm the docs site does not block AI crawlers, then lay out the pricing page, OpenAPI spec, status page and sample code. On 2026-11-09 we read the robots.txt of 12 well-known API product docs sites: none blocks GPTBot, ClaudeBot or PerplexityBot by name, and the main docs are all Allow; only Twilio writes ai-train=no in its Content-Signal. The ones we checked publish OpenAPI files on GitHub, and all 8 products we checked have a public status page. The pricing page should state the billing unit, a usage tier table, rate limits and an update date. Whether AI reads specs and status pages, and what it cites when recommending an API, is not stated officially and we have not verified it.

Do API products' docs sites block AI crawlers?

Among the 12 well-known API products we checked, none blocks GPTBot, ClaudeBot or PerplexityBot by name. We read their robots.txt files on 2026-11-09: the main docs are Allow for all three crawlers, and what is blocked is mostly console, search, internal or parameter paths. Five of them put a Content-Signal line in robots.txt, and only Twilio sets ai-train=no.

Product (docs site)GPTBotClaudeBotPerplexityBotNotable in robots.txt
Stripe (docs.stripe.com)AllowAllowAllowContent-Signal: ai-train=yes, search=yes, ai-input=yes
OpenAI (developers.openai.com)AllowAllowAllowAllow: /
Anthropic (platform.claude.com)AllowAllowAllowDisallow: /api/
Twilio (www.twilio.com/docs)AllowAllowAllowContent-Signal: ai-train=no, search=yes, ai-input=yes
Cloudflare (developers.cloudflare.com)AllowAllowAllowContent-Signal: ai-train=yes, search=yes, ai-input=yes
GitHub (docs.github.com)AllowAllowAllowUser-agent: * ; no rules
Resend (resend.com/docs)AllowAllowAllowContent-Signal: ai-train=yes, search=yes, ai-input=yes
Supabase (supabase.com/docs)AllowAllowAllowContent-Signal: ai-train=yes, search=yes, ai-input=yes
Shopify (shopify.dev)Allow (named block)Allow (named block)Allow (via *)GPTBot, ClaudeBot, ChatGPT-User, Claude-User, CCBot named; same Disallow paths
Plaid (plaid.com/docs)AllowAllowAllowDisallow: /api/* /legacy-docs/ /sandbox/* *.pdf
Postman (www.postman.com)Allow (named block)Allow (named block)Allow (named block)Allow: / ; Disallow: /search?* ; llms.txt linked in comments
Mapbox (docs.mapbox.com)AllowAllowAllowDisallow: /archive/

How to read it: plain Allow or Allow (via *) means the crawler is not named and follows the User-agent: * rules; named block means it is named and has its own rules. The judgment covers docs paths only (for example /docs/…), is inferred from the text of robots.txt, and is not the behavior of real crawlers; a CDN or firewall may block separately, see How do you write robots.txt for AI crawlers?. For crawler names and purposes see AI crawler user agents: a side-by-side table.

Content-Signal is a line in robots.txt that states whether content may be used for model training (ai-train), search (search) and AI answer input (ai-input). Twilio allows search and AI answer input but not training; this is a stance it declares itself and does not change the Allow or Disallow result.

What should a pricing page say so AI treats you as a fitting API to recommend?

When users ask "which email API fits me", the answer compares price, usage tiers, rate limits and supported languages. All of these must be in the body text of the pricing page. For the general way to write a pricing page see How should a pricing page be written so AI answers correctly?; an API product needs the following in addition.

What to writeHow to write itWhy
Billing unitPer thousand requests, per message, per million tokens, and so on; write the number and currencyAI needs one common unit to compare different APIs
Usage tier tableMonthly usage per tier, unit price and the price beyond the tier, as a tableA table is easier to read than an image or a slider
Rate limitsRequests per minute or per second, listed by tierA hard criterion when developers choose an API
SLA and supportAvailability commitment, support channels and response time, stating which tier has themAnswers "is it fit for production?"
Update dateWrite on the page: "Pricing updated 2026-11"The most common reason AI gets prices wrong is reading an old page

Do not put prices only in a click-to-switch component or an image; fetch the page with curl and confirm the price numbers are in the returned HTML.

Should the OpenAPI spec be public?

We recommend making it public and findable from both the docs and the repository. All the products we checked put a public OpenAPI file on GitHub, and all of them opened on 2026-11-09. A spec is a machine-readable list of the interface: paths, parameters, response fields and authentication, which AI coding tools and search engines can read directly instead of guessing from doc pages. Whether AI answers read these files is not stated officially and we have not verified it.

ProductWhere the public spec livesWhat we saw (2026-11-09)
Stripegithub.com/stripe/openapispec3.json opens
OpenAIgithub.com/openai/openai-openapiopenapi.yaml, openapi: 3.1.0
Twiliogithub.com/twilio/twilio-oaispec/json/twilio_api_v2010.json opens
GitHubgithub.com/github/rest-api-descriptionapi.github.com.json, openapi 3.0.3
Plaidgithub.com/plaid/plaid-openapi2020-09-14.yml, openapi 3.0.0
Cloudflaregithub.com/cloudflare/api-schemasrepository page opens
Resendgithub.com/resend/resend-openapirepository page opens

We only confirmed that the addresses open; we did not check whether these specs are current or complete. What you can do yourself:

  1. 1. Put the spec at a fixed address

    Keep one file in the repository, link to it from the docs site, and keep the address stable.

  2. 2. Fill in descriptions in the spec

    One sentence per endpoint saying what it does; for each parameter give the unit and range of values.

  3. 3. Link to the spec from the docs

    At the top of the "API reference" page write "OpenAPI spec: URL", and include it in the sitemap.

  4. 4. Update it with every release

    The version number and date in the spec should match the changelog (see Do changelog pages help AI search?).

What should the status page and changelog contain?

A status page answers "is this API stable?" and a changelog answers "is it still maintained?". All 8 products we checked have a public status page, and all opened on 2026-11-09:

ProductStatus page
Stripestatus.stripe.com
OpenAIstatus.openai.com
Anthropicstatus.claude.com (also /api/v2/status.json, which returns a very short JSON)
Twiliostatus.twilio.com
Cloudflarewww.cloudflarestatus.com
GitHubwww.githubstatus.com (also /api/v2/status.json)
Resendresend-status.com
Supabasestatus.supabase.com

How: host the status page at status.your-domain; link to it from the site footer and the docs footer; link the SLA line on the pricing page to it; keep past incidents as readable text, not only charts. For changelogs see Do changelog pages help AI search?.

How should sample code be written so AI can cite it?

Give each endpoint one curl example that can be copied and run, then SDK examples in the main languages. Put the code in the HTML, not in images or in tabs that load only after a click.

curl https://api.example.com/v1/emails \
  -H "Authorization: Bearer $EXAMPLE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"from": "hello@example.com", "to": "you@example.org", "subject": "Hi", "text": "Hello"}'
  1. 1. Give both request and response

    The full request command plus a returned JSON, with field names that match the spec.

  2. 2. State the install command and version

    For example the npm or pip install command and the list of languages the SDK supports.

  3. 3. One task per page

    Pages like "send an email with Node.js" or "how to verify a webhook signature", answering directly in the opening. See How to write use-case pages.

  4. 4. Write error codes as a table

    Status code, meaning and how to handle it.

How do you get AI to mention you in "recommend an API" answers?

Besides the pages above, comparison questions usually cite comparison pages, review articles and developer community discussions.

  1. 1. First confirm your site does not block crawlers

    Use AI crawler check to look at robots.txt, then check the CDN's bot rules.

  2. 2. Write 3 comparison pages

    "Your API vs the two it is most often compared with", with a source and date for each item; see How to write comparison pages.

  3. 3. Write "best for whom" use-case pages

    One scenario per page, such as "order notifications for online shops".

  4. 4. Add WebAPI structured data (optional)

    Keep fields consistent with the body text; which AIs read it is not stated officially, see Does structured data help AI search?.

  5. 5. Ask AI again in a month

    Ask "which API for scenario X" and see whether you are mentioned and which URLs are cited.

{
  "@context": "https://schema.org",
  "@type": "WebAPI",
  "name": "Example Mail API",
  "description": "REST API for sending transactional email; JSON over HTTPS.",
  "documentation": "https://example.com/docs/",
  "termsOfService": "https://example.com/terms/",
  "provider": { "@type": "Organization", "name": "Example Inc.", "url": "https://example.com/" }
}

For how to write the docs site itself and whether to make llms.txt, see How do developer docs get cited by AI search? and What is llms.txt?; for the general recommendation questions of SaaS see How do SaaS products get recommended by ChatGPT?.

FAQ

Can AI crawlers read API docs sites?

Judging by robots.txt, yes: on 2026-11-09 we read 12 well-known API product docs sites and none blocks GPTBot, ClaudeBot or PerplexityBot by name. This is a judgment from the text of robots.txt; a CDN or firewall may block separately.

What is Content-Signal?

It is a line in robots.txt that says yes or no for ai-train (training), search (search) and ai-input (AI answer input). Stripe, Cloudflare, Resend and Supabase have yes for all three; Twilio has ai-train=no, search=yes, ai-input=yes (2026-11-09).

Will an OpenAPI spec make AI recommend my API?

We found no official statement and no actual evidence, so this is not verified. The benefit of a public spec is that it is machine-readable and AI coding tools can use it directly; the effect on "which API to recommend" is uncertain.

How should a pricing page be written so AI does not get it wrong?

Write the price, billing unit, usage tiers and update date as text and tables in the body, not only in images or click-to-switch components. Fetch the page with curl and confirm the price numbers are in the HTML.

Does a status page help AI recommendations?

There is no official statement, so this is not verified. At least it answers "is this API stable?"; all 8 products we checked have a public status page, and some offer a very short JSON endpoint.

Sources

kagentx

Can crawlers read your pages?
Let the evaluation tell you

The evaluation checks whether crawlers can get in and whether the text is served directly, then asks ChatGPT, Perplexity and Claude (with web search on) your buyers' questions and sees whether you are mentioned. The Basic plan is $1, and results usually take 2–3 minutes.