Guide · Crawlers

Will AI crawlers crawl too much? Which crawlers honor Crawl-delay

Updated 2026-11-07 · kagentx

Short answer

Crawl-delay is not a universal switch: Anthropic's documentation says it is supported, Google's and Apple's say it is not, the OpenAI, Perplexity, Common Crawl and Meta pages do not mention it, and we could not read Bing's original text, so that is marked not verified. For a small site the more common problem is AI crawlers not coming or not crawling everything, not crawling too much; count each crawler's requests in your own server logs first, and throttle only when responses slow down or you see 5xx.

Which AI crawlers honor Crawl-delay in robots.txt?

Only Anthropic's documentation explicitly says "supported"; Google and Apple explicitly say "not supported"; the OpenAI, Perplexity, Common Crawl and Meta pages do not mention it; we could not read Bing's original text. Below is what we opened page by page on 2026-11-07. Where a document is silent, that is the document being silent, not our guess that the crawler ignores it.

Crawler / companyWhat the official documentation says about Crawl-delayPage we read
ClaudeBot (Anthropic)Supported: "we support the non-standard Crawl-delay extension", example Crawl-delay: 1; "respecting Crawl-delay where appropriate"Crawler explanation on support.claude.com
Googlebot (Google)Not supported: the supported fields are only user-agent, allow, disallow and sitemap, and crawl-delay is explicitly stated as "aren't supported"Google's robots.txt specification page
Applebot (Apple)Not followed: "Applebot does not follow crawl-delay"; the crawl rate adjusts automatically when a site slows down or returns errorssupport.apple.com/119829
GPTBot, OAI-SearchBot, ChatGPT-User (OpenAI)The page mentions neither Crawl-delay nor a rate; ChatGPT-User is user-triggered, and robots.txt rules "may not apply"developers.openai.com crawler page
PerplexityBot, Perplexity-UserThe page mentions neither Crawl-delay nor crawl speed; it has IP lists, and robots.txt changes take up to 24 hoursdocs.perplexity.ai crawler page
CCBot (Common Crawl)The page mentions neither Crawl-delay nor a rate; it uses dedicated IP ranges and reverse DNScommoncrawl.org/ccbot
meta-externalagent (Meta)The page covers only Allow / Disallow, with no Crawl-delay or rate settingMeta's web crawlers page
Bingbot (Microsoft)Not verified: we only read the title of Bing's help pages; we saw the claim that it honors Crawl-delay only in a search-result summary, not in the original text—

The takeaway is simple: Crawl-delay is not a universal switch. Only Anthropic states in black and white that it honors it; you cannot manage every AI crawler with one Crawl-delay line. The full list of crawlers and their purposes is in AI crawler user agents: a side-by-side table, and robots.txt in general is in How to write robots.txt for AI crawlers.

User-agent: ClaudeBot
Crawl-delay: 1

# Anthropic documents this one. For other crawlers it may simply be ignored.

Do AI crawlers really crawl small sites too often?

We have no statistics on this, and no official document gives a concrete crawl rate, so we do not conclude "they don't". What we can say is the common failure direction: a new product site has few pages, and the more common problem is that crawlers have not come, or came and did not crawl everything, rather than crawling too much. Judge by your own logs: count each crawler's requests first, then decide whether to manage it.

# How many requests did each AI crawler make in your access log? (Common Log / nginx combined format)
grep -E "GPTBot|OAI-SearchBot|ClaudeBot|Claude-SearchBot|PerplexityBot|CCBot|Applebot" access.log \
  | awk '{print $1}' | sort | uniq -c | sort -rn | head

When you need to act: one crawler's request volume in a short time is high enough to affect normal visits (slow responses, 5xx). Then go in this order: confirm it is the real crawler (check against each provider's published IP list); for crawlers that honor Crawl-delay, write Crawl-delay first; for the rest, rate-limit at the CDN.

The opposite problem, how long until a new page is crawled and cited, is in How long until AI cites a new page; telling search engines about new pages proactively is in what IndexNow is.

How do other sites write Crawl-delay?

On 2026-11-07 we read several robots.txt files with curl to see how others use it.

SiteCrawl-delay in robots.txt
news.ycombinator.comCrawl-delay: 30 under the general rule (User-Agent: *)
aiweekly.beehiiv.com (beehiiv's default robots.txt)AhrefsBot and AhrefsSiteAudit each have Crawl-delay: 10; Amazonbot is disallowed site-wide; no Crawl-delay for AI crawlers
kagentx.com, mochistamp.com, pagent.kagentx.comNone has Crawl-delay
gammaslide.comOnly one comment line mentions Crawl-delay; there is no directive

This does not conclude that Crawl-delay works. It only shows that big sites and platforms write it for individual SEO tools, that AI crawlers mostly have no separate throttle, and that none of our own four sites writes it either.

How do I limit crawler request speed with Cloudflare?

Cloudflare's rate limiting rules documentation (page dated 2026-08-25) says every plan from the entry-level tier up to Enterprise has them. On the entry-level plan you can match on path and "Verified Bot", with only 1 rule, a 10-second counting period and mitigation timeout, counted by IP only; matching on User Agent needs the Business plan. So a small site on the entry-level plan cannot limit "only GPTBot"; it can only limit a path or all verified bots.

The AI Crawl Control documentation (updated 2026-08-14) covers allowing or blocking individual crawlers and does not mention rate limiting. How to set it is in Does Cloudflare block AI crawlers by default? How to check and allow them.

What should a small site do?

  1. 1. Allow first, do not rush to throttle

    If you have not been crawled yet, first make sure robots.txt and the CDN are not keeping crawlers out; see How to write robots.txt for AI crawlers and Does Cloudflare block AI crawlers by default? How to check and allow them.

  2. 2. Count the logs

    Use the command above to see how many times each AI crawler comes per day. If you have no logs, turn logging on first.

  3. 3. Throttle only when something is wrong

    When responses slow down or you see 5xx, first check it is the real crawler, then add one line for crawlers that honor Crawl-delay and rate-limit the rest at the CDN.

  4. 4. Look again in two weeks

    After the change, check whether that crawler's request count in the logs and your server response time have recovered.

How to work through a page that Google marks “Discovered – currently not indexed”: see “Discovered – currently not indexed”: what to do? Discovered vs Crawled.

Whether a page with nothing unique should be noindexed, and what noindex and robots.txt Disallow each do for Google and for AI crawlers: see Should thin pages be noindexed? What it means for Google and AI search.

FAQ

Can AI crawlers overwhelm a small website?

We have no statistics on this, and no official document gives a concrete crawl rate. For a product site with tens to hundreds of pages, the common problem is crawlers not coming or not crawling everything, not crawling too much; check real request counts in your server logs first.

Does Crawl-delay in robots.txt help?

Only for crawlers that honor it. Anthropic's documentation says it supports this non-standard extension; Google's says it does not; Apple's says Applebot does not follow it. The OpenAI, Perplexity, Common Crawl and Meta pages do not mention it, so do not assume they honor it.

Does Bingbot honor Crawl-delay?

We could not read the body of Bing's help pages and only saw the "honors it" claim in a search-result summary, so it is marked not verified. Bing Webmaster Tools has Crawl Control, which lets you adjust crawl speed in the dashboard.

Can Cloudflare limit how fast AI crawlers crawl?

Yes, with rate limiting rules. The documentation (updated 2026-08-25) says the entry-level plan can match on path and Verified Bot, with only 1 rule; matching on User Agent needs the Business plan. The AI Crawl Control documentation covers only allow or block, not rate limits.

How do I know which AI crawlers actually came?

Look at your server or CDN access logs, count requests by User-Agent, then check against each provider's published IP list that they are not impersonators.

Sources

kagentx

Can crawlers read your pages?
Let the evaluation tell you

The evaluation checks whether crawlers can get in and whether the text is served directly, then asks ChatGPT, Perplexity and Claude (with web search on) your buyers' questions and sees whether you are mentioned. The Basic plan is $1, and results usually take 2–3 minutes.