Guide · Status codes

How do HTTP status codes affect AI crawlers? 404, 410, 301, 429, 5xx

Updated 2026-11-09 · kagentx

Short answer

Google’s official documentation handles 404 and 410 the same way, treats 301 as a strong signal and 302 as a weak one, slows crawling temporarily on 429 and 5xx, and drops pages from the index after persistent 5xx. The OpenAI and Perplexity pages we read say nothing about status codes, so we write “not stated officially”. On 2026-11-09 we tested a non-existent address on four sites with curl: all returned a real 404, with no soft 404.

What does Google’s official documentation say about each status code?

This section comes entirely from Google’s “HTTP status codes, and network and DNS errors” documentation (read 2026-11-09; the page shows no update date). It describes Google Search’s crawler, not AI crawlers.

Status codeWhat Google’s documentation says
200, 201Passed to the next processing step; “indexing is not guaranteed”
204No content, nothing to process
2xx but the page looks like an errorSearch Console reports a soft 404
301, 308Followed; strong signal that the target is the URL to process
302, 303, 307Followed; weak signal
304Unchanged since the last crawl
404, 410 (and 400, 401, 403, 411)Treated as content that does not exist; indexed URLs are removed; newly found 404 pages are not processed; crawl frequency gradually decreases
429Treated as a sign of an overloaded server and handled as a 5xx
5xx and 429Crawl rate temporarily slows and gradually rises again after 2xx responses return; indexed URLs are kept for now and eventually dropped
500, 502, 503Crawl rate drops in proportion to the number of failing URLs; URLs that persistently return a server error are removed from the index

Three more points: Googlebot generally follows up to 10 redirect hops; 401 and 403 should not be used to limit crawl rate; and the documentation gives no number of days for how long “persistently” returning errors lasts.

What do OpenAI’s and Perplexity’s official documents say?

We read both companies’ crawler pages. Neither mentions status codes, redirects or crawling behavior on errors:

ProductPage we readWhat it says about status codes
ChatGPT search (OAI-SearchBot)OpenAI crawler page at developers.openai.com, read 2026-11-09Nothing; it only says sites that opt out will not appear in ChatGPT search answers and that robots.txt changes take about 24 hours to apply. Status code handling: not stated officially
Perplexity (PerplexityBot)Perplexity crawler page at docs.perplexity.ai, read 2026-11-09Nothing. Status code handling: not stated officially (only this one page was checked)

So for AI crawlers all we can say is: setting status codes the way Google describes is the safest approach because it matches the general meaning of HTTP; what a given AI crawler does on a 404 or 429 is not stated in any official document. For a list of crawlers see AI crawler user agents: a side-by-side table.

What do non-existent addresses on our four sites return?

On 2026-11-09 we used curl to request one address that certainly does not exist, /this-page-does-not-exist-20261109/, on each site, and also a non-existent .pdf file. We requested each once with an ordinary browser user agent, GPTBot/1.1, PerplexityBot/1.0 and Googlebot/2.1; the status codes on all four sites were the same regardless of user agent.

SiteNon-existent addressResponse type and sizePage title or content
kagentx.com404Empty body, 0 bytesNone
mochistamp.com404application/json, 63 bytes{"code":404,"message_key":"requestFailed","detail":"Not Found"}
gammaslide.com404text/html, 3,939 bytesPage not found | Gamma AI Slide
pagent.kagentx.com404text/html, 5,474 bytesNot found · Pagent

None of the four sites has a soft 404: non-existent addresses return a real 404, not a 200 with a “not found” page. The difference is the body: kagentx.com returns an empty body, mochistamp.com returns a piece of JSON, and gammaslide.com and pagent.kagentx.com return an HTML error page with a title. For a crawler the status code says everything, but for a person an error page with navigation is more useful.

Other results:

CheckResult
http:// requests to the four home pagesAll return 301 to https
www.kagentx.com and www.gammaslide.comBoth return 301 to the domain without www
kagentx.com/guides (no trailing slash)307 (temporary redirect) to /guides/; this is a to-do for us: a permanent add-the-slash redirect should be a 301
kagentx.com/p/zz-nope/ (a non-existent product page)404 with a 155,268-byte body (the product library shell page; the status code is correct)
Non-existent addresses under /agents/, /zh/ and /guides/ on kagentx.com404, 0 bytes
mochistamp.com/en/guide/size200

This round tested only non-existent addresses and redirects, not 5xx or 429: we have no safe way to produce those two states on a live site.

How do you check whether your site returns soft 404s?

curl -s -o /dev/null -w "%{http_code} %{content_type} %{size_download}\n" \
  -A "GPTBot/1.1" https://example.com/this-page-does-not-exist/
# expected: 404 text/html 3939   (a real error page with a real 404 status)
# warning:  200 text/html 3939   (the same page served as 200 = a soft 404)

Request an address that certainly does not exist, once with an ordinary user agent and once with an AI crawler’s user agent, and look at the status code. An error page that returns 200 is a soft 404 and should be fixed.

Which status code should you use in which situation?

SituationUse thisReason (source)
A page has moved permanently to a new address301 (or 308)Google: strong signal that the target is the URL to process
A temporary redirect (campaign page, maintenance page)302 or 307Google: weak signal
A page was deleted with no replacement404 or 410Google: handled the same way; indexed URLs are removed
You want crawlers to slow down429 or 503Google: crawl rate slows temporarily and gradually rises again
You want to keep a certain crawler outrobots.txt or a firewall rule; do not use 401 or 403 to limit rateGoogle: 401 and 403 should not be used to limit crawl rate
Server failure5xx, restored as quickly as possibleGoogle: URLs that persistently return a server error are removed from the index

For the full migration process see How do you keep search and AI citations when you change domain or URLs?; for robots.txt see How do you write robots.txt for AI crawlers?; for rate-limiting by crawler on Cloudflare see Does Cloudflare block AI crawlers by default? How to check and allow them; deleted PDFs work the same way, and the other PDF rules are in Do AI answers cite PDF whitepapers? Should you put content in a PDF.

How do you confirm which status codes crawlers actually got?

  1. 1. Test an address that does not exist with curl

    Test with several user agents and confirm you get a real 404, not a 200.

  2. 2. Read the server logs

    Filter for crawler requests and count by status code. Which pages return 404, 429 or 5xx is then obvious; for the method see How do I check whether GPTBot and other AI crawlers have visited my site?.

  3. 3. Check redirect chains

    Use 301 for permanent moves and keep chains short; Google generally follows up to 10 hops.

  4. 4. robots.txt itself must be reachable

    Google’s documentation says a 5xx on robots.txt makes it stop crawling for 12 hours first; a 4xx (except 429) is treated as having no robots.txt.

  5. 5. Understand crawl budget

    Too many errors waste crawling; see Will AI crawlers crawl too much? Which crawlers honor Crawl-delay.

Whether a URL missing its trailing slash gets a 307 or a 301, and how Google sees it: see Is a 307 on trailing slashes bad for SEO? How to choose between 301, 302, 307 and 308.

FAQ

Is there a difference between 404 and 410 for search?

Google’s documentation groups 404 and 410 and handles them the same way: the content is treated as nonexistent, indexed URLs are removed and newly found 404 pages are not processed. You do not need to switch to 410 to tell crawlers a page is gone for good. Whether AI crawlers distinguish them: not stated officially.

What is a soft 404?

A page that is really an error page, an empty page or a “not found” message but returns 200. Google’s documentation says that when a 2xx page looks like an error, Search Console reports a soft 404. The fix is to return a real 404 or 410, or redirect to a suitable page.

Should I use 301 or 302?

Use 301 (308 is equivalent) when an address has moved permanently; Google treats it as a strong signal that the target is the URL to process. For temporary redirects use 302 or 307, a weak signal. Using a temporary redirect for a permanent move sends a weaker signal. Whether AI crawlers treat them the same way: not stated officially.

What does a crawler do on a 429?

Google treats 429 as a sign of an overloaded server and handles it like a 5xx: crawl rate slows temporarily and gradually rises once 2xx responses return. To limit crawlers use 429 or 503, not 401 or 403; Google’s documentation says 4xx codes other than 429 do not affect crawl rate. How other crawlers handle 429: not stated officially.

Will server 5xx errors drop me from the index?

Occasional ones will not. Google’s documentation says indexed URLs are kept at first and URLs that persistently return a server error are removed; how long “persistently” is is not written down. So restore service quickly after an outage and check the logs to see what crawlers received.

Sources

kagentx

Can crawlers read your pages?
Let the evaluation tell you

The evaluation checks whether crawlers can get in and whether the text is served directly, then asks ChatGPT, Perplexity and Claude (with web search on) your buyers' questions and sees whether you are mentioned. The Basic plan is $1, and results usually take 2–3 minutes.