What does Google’s official documentation say about each status code?
This section comes entirely from Google’s “HTTP status codes, and network and DNS errors” documentation (read 2026-11-09; the page shows no update date). It describes Google Search’s crawler, not AI crawlers.
| Status code | What Google’s documentation says |
|---|---|
| 200, 201 | Passed to the next processing step; “indexing is not guaranteed” |
| 204 | No content, nothing to process |
| 2xx but the page looks like an error | Search Console reports a soft 404 |
| 301, 308 | Followed; strong signal that the target is the URL to process |
| 302, 303, 307 | Followed; weak signal |
| 304 | Unchanged since the last crawl |
| 404, 410 (and 400, 401, 403, 411) | Treated as content that does not exist; indexed URLs are removed; newly found 404 pages are not processed; crawl frequency gradually decreases |
| 429 | Treated as a sign of an overloaded server and handled as a 5xx |
| 5xx and 429 | Crawl rate temporarily slows and gradually rises again after 2xx responses return; indexed URLs are kept for now and eventually dropped |
| 500, 502, 503 | Crawl rate drops in proportion to the number of failing URLs; URLs that persistently return a server error are removed from the index |
Three more points: Googlebot generally follows up to 10 redirect hops; 401 and 403 should not be used to limit crawl rate; and the documentation gives no number of days for how long “persistently” returning errors lasts.
What do OpenAI’s and Perplexity’s official documents say?
We read both companies’ crawler pages. Neither mentions status codes, redirects or crawling behavior on errors:
| Product | Page we read | What it says about status codes |
|---|---|---|
| ChatGPT search (OAI-SearchBot) | OpenAI crawler page at developers.openai.com, read 2026-11-09 | Nothing; it only says sites that opt out will not appear in ChatGPT search answers and that robots.txt changes take about 24 hours to apply. Status code handling: not stated officially |
| Perplexity (PerplexityBot) | Perplexity crawler page at docs.perplexity.ai, read 2026-11-09 | Nothing. Status code handling: not stated officially (only this one page was checked) |
So for AI crawlers all we can say is: setting status codes the way Google describes is the safest approach because it matches the general meaning of HTTP; what a given AI crawler does on a 404 or 429 is not stated in any official document. For a list of crawlers see AI crawler user agents: a side-by-side table.
What do non-existent addresses on our four sites return?
On 2026-11-09 we used curl to request one address that certainly does not exist, /this-page-does-not-exist-20261109/, on each site, and also a non-existent .pdf file. We requested each once with an ordinary browser user agent, GPTBot/1.1, PerplexityBot/1.0 and Googlebot/2.1; the status codes on all four sites were the same regardless of user agent.
| Site | Non-existent address | Response type and size | Page title or content |
|---|---|---|---|
| kagentx.com | 404 | Empty body, 0 bytes | None |
| mochistamp.com | 404 | application/json, 63 bytes | {"code":404,"message_key":"requestFailed","detail":"Not Found"} |
| gammaslide.com | 404 | text/html, 3,939 bytes | Page not found | Gamma AI Slide |
| pagent.kagentx.com | 404 | text/html, 5,474 bytes | Not found · Pagent |
None of the four sites has a soft 404: non-existent addresses return a real 404, not a 200 with a “not found” page. The difference is the body: kagentx.com returns an empty body, mochistamp.com returns a piece of JSON, and gammaslide.com and pagent.kagentx.com return an HTML error page with a title. For a crawler the status code says everything, but for a person an error page with navigation is more useful.
Other results:
| Check | Result |
|---|---|
| http:// requests to the four home pages | All return 301 to https |
| www.kagentx.com and www.gammaslide.com | Both return 301 to the domain without www |
| kagentx.com/guides (no trailing slash) | 307 (temporary redirect) to /guides/; this is a to-do for us: a permanent add-the-slash redirect should be a 301 |
| kagentx.com/p/zz-nope/ (a non-existent product page) | 404 with a 155,268-byte body (the product library shell page; the status code is correct) |
| Non-existent addresses under /agents/, /zh/ and /guides/ on kagentx.com | 404, 0 bytes |
| mochistamp.com/en/guide/size | 200 |
This round tested only non-existent addresses and redirects, not 5xx or 429: we have no safe way to produce those two states on a live site.
How do you check whether your site returns soft 404s?
curl -s -o /dev/null -w "%{http_code} %{content_type} %{size_download}\n" \
-A "GPTBot/1.1" https://example.com/this-page-does-not-exist/
# expected: 404 text/html 3939 (a real error page with a real 404 status)
# warning: 200 text/html 3939 (the same page served as 200 = a soft 404)
Request an address that certainly does not exist, once with an ordinary user agent and once with an AI crawler’s user agent, and look at the status code. An error page that returns 200 is a soft 404 and should be fixed.
Which status code should you use in which situation?
| Situation | Use this | Reason (source) |
|---|---|---|
| A page has moved permanently to a new address | 301 (or 308) | Google: strong signal that the target is the URL to process |
| A temporary redirect (campaign page, maintenance page) | 302 or 307 | Google: weak signal |
| A page was deleted with no replacement | 404 or 410 | Google: handled the same way; indexed URLs are removed |
| You want crawlers to slow down | 429 or 503 | Google: crawl rate slows temporarily and gradually rises again |
| You want to keep a certain crawler out | robots.txt or a firewall rule; do not use 401 or 403 to limit rate | Google: 401 and 403 should not be used to limit crawl rate |
| Server failure | 5xx, restored as quickly as possible | Google: URLs that persistently return a server error are removed from the index |
For the full migration process see How do you keep search and AI citations when you change domain or URLs?; for robots.txt see How do you write robots.txt for AI crawlers?; for rate-limiting by crawler on Cloudflare see Does Cloudflare block AI crawlers by default? How to check and allow them; deleted PDFs work the same way, and the other PDF rules are in Do AI answers cite PDF whitepapers? Should you put content in a PDF.
How do you confirm which status codes crawlers actually got?
1. Test an address that does not exist with curl
Test with several user agents and confirm you get a real 404, not a 200.
2. Read the server logs
Filter for crawler requests and count by status code. Which pages return 404, 429 or 5xx is then obvious; for the method see How do I check whether GPTBot and other AI crawlers have visited my site?.
3. Check redirect chains
Use 301 for permanent moves and keep chains short; Google generally follows up to 10 hops.
4. robots.txt itself must be reachable
Google’s documentation says a 5xx on robots.txt makes it stop crawling for 12 hours first; a 4xx (except 429) is treated as having no robots.txt.
5. Understand crawl budget
Too many errors waste crawling; see Will AI crawlers crawl too much? Which crawlers honor Crawl-delay.
Whether a URL missing its trailing slash gets a 307 or a 301, and how Google sees it: see Is a 307 on trailing slashes bad for SEO? How to choose between 301, 302, 307 and 308.
FAQ
Is there a difference between 404 and 410 for search?
Google’s documentation groups 404 and 410 and handles them the same way: the content is treated as nonexistent, indexed URLs are removed and newly found 404 pages are not processed. You do not need to switch to 410 to tell crawlers a page is gone for good. Whether AI crawlers distinguish them: not stated officially.
What is a soft 404?
A page that is really an error page, an empty page or a “not found” message but returns 200. Google’s documentation says that when a 2xx page looks like an error, Search Console reports a soft 404. The fix is to return a real 404 or 410, or redirect to a suitable page.
Should I use 301 or 302?
Use 301 (308 is equivalent) when an address has moved permanently; Google treats it as a strong signal that the target is the URL to process. For temporary redirects use 302 or 307, a weak signal. Using a temporary redirect for a permanent move sends a weaker signal. Whether AI crawlers treat them the same way: not stated officially.
What does a crawler do on a 429?
Google treats 429 as a sign of an overloaded server and handles it like a 5xx: crawl rate slows temporarily and gradually rises once 2xx responses return. To limit crawlers use 429 or 503, not 401 or 403; Google’s documentation says 4xx codes other than 429 do not affect crawl rate. How other crawlers handle 429: not stated officially.
Will server 5xx errors drop me from the index?
Occasional ones will not. Google’s documentation says indexed URLs are kept at first and URLs that persistently return a server error are removed; how long “persistently” is is not written down. So restore service quickly after an outage and check the logs to see what crawlers received.
Sources
- Google · HTTP status codes, and network and DNS errors (Checked 2026-11-09; the page shows no update date)
- Google · How Google interprets the robots.txt specification (Checked 2026-11-09; the page shows no update date)
- OpenAI · Overview of OpenAI Crawlers (Checked 2026-11-09; the page shows no update date; the page does not cover status codes)
- Perplexity · Perplexity Crawlers (Checked 2026-11-09; the page shows no update date; the page does not cover status codes)
- kagentx evaluations: the data on the case study pages; each example gives its date.