What do Applebot and Applebot-Extended each control?
Applebot is the crawler that fetches web pages; Applebot-Extended is only a marker in robots.txt. Key points from Apple's official page About Applebot (opened and checked on 2026-11-09; the page shows a publication date of 2026-09-04):
| Name | What Apple's page says | What it means for you |
|---|---|---|
| Applebot | Crawled data powers the search technology in Spotlight, Siri and Safari; allow it in robots.txt and your content may appear in search results for Apple users | To be found by Apple's search, do not block it |
| Applebot data used for training | The data may also help train Apple's foundation models, including those behind Apple Intelligence | To keep content out of training, use Applebot-Extended |
| Applebot data used in answers | The data may supply extra context and up-to-date content when AI models generate output, for example answering broad world-knowledge questions in Siri and Search, possibly with links to sources | This is Apple's own wording for Siri's web answers; which pages get used is not stated |
| Applebot-Extended | A secondary user agent: it does not crawl pages and only decides whether data fetched by Applebot may be used for training; pages that disallow it can still appear in search results; its rules are not used in Search ranking | Blocking it affects training only, not whether you can be found |
For the names and purposes of the other AI crawlers, see AI crawler user agents: a side-by-side table.
If you block Applebot-Extended, can Siri and Spotlight still find you?
Yes. What Apple's page says, in short: even if you disallow Applebot-Extended and add nosnippet to content, your site's rules may still let Applebot crawl the pages, and the content remains discoverable through Spotlight, Siri and Safari. What decides whether you can be found is whether Applebot can crawl, and whether the page has noindex.
| What you do | Training | Background content in Siri and similar answers | Findable in Spotlight, Siri and Safari |
|---|---|---|---|
| Write nothing | May be used | May be used | Yes |
| disallow Applebot-Extended | Opted out | May still be used (the page treats them separately) | Yes |
| Add nosnippet to a page | Unchanged | Not used; suggestions show only the title | Yes |
| Add noindex to a page | Unchanged | Not shown | Not shown in Spotlight or Siri suggestions |
| disallow Applebot | Cannot crawl | Cannot crawl | The page does not spell out each consequence; we read it as "cannot crawl" and have not verified the finer cases |
The first four rows come from Apple's text; the last row is our reading: the page says allowing Applebot is what lets content appear in search results.
How should robots.txt be written?
To be recommended you need to write nothing; add Applebot-Extended only if you want to opt out of training. Like this:
# Default: say nothing about Applebot. Apple's page says that if robots.txt # does not mention Applebot but does mention Googlebot, Applebot follows the Googlebot rules. # Opt out of training only (Apple: Applebot-Extended does not crawl pages) User-agent: Applebot-Extended Disallow: / # Keep Applebot itself allowed so pages can still appear in Spotlight, Siri and Safari User-agent: Applebot Allow: /
Three things to note:
1. You did not mention Applebot but did mention Googlebot
Apple's page says Applebot then follows the Googlebot rules, so a Disallow you added for Googlebot also applies to Applebot.
2. Do not block scripts and styles
The page says Applebot may render pages in a browser, and if JS, CSS or XHR are blocked in robots.txt the page may not render properly. For how crawlers handle scripts in general, see Do AI crawlers render JavaScript?.
3. It ignores crawl-delay
The page says Applebot does not follow crawl-delay; it slows itself down when your site gets slow or returns errors.
To keep a single page out of answers, use nosnippet:
<!-- One page: no description or web answer is generated for it. Apple: the page then shows only its title. --> <meta name="robots" content="nosnippet"> # Same thing as an HTTP header, for PDFs and other non-HTML files X-Robots-Tag: applebot: nosnippet
Apple's page states the cost of nosnippet: Applebot no longer generates a description or web answer for the page, and suggestions show only the title. Do not add it to pages you want recommended. For the general syntax and other crawlers, see How do you write robots.txt for AI crawlers?.
Do our four sites' robots.txt files name Applebot?
None names Applebot, and none names Googlebot either, so the default User-agent: * rules apply. On 2026-11-09 we read the robots.txt of our four sites with curl, then fetched each home page with an ordinary browser identity and with Applebot's User-Agent string:
| Site | robots.txt | Names Applebot or Googlebot | Paths disallowed under * | Home page (ordinary / Applebot UA) |
|---|---|---|---|---|
| mochistamp.com | 200 | None | /api/(放行 /api/content.js 和 /api/strings/)、/files/、/admin.html | 200 / 200 |
| gammaslide.com | 200 | None | /d/、/decks、/account、/admin | 200 / 200 |
| kagentx.com | 200 | None | /admin.html、/candidates.html、/api/(其余 Allow: /) | 200 / 200 |
| pagent.kagentx.com | 200 | None | /i18n/ | 200 / 200 |
How to read it: what is blocked is back-office, API and account pages; article and product pages are not blocked. This only checks the robots.txt text and a User-Agent string, not the behaviour of Apple's real crawler: our network is in mainland China, the IP is outside Apple's ranges, and we have not searched for these four sites in Siri or Spotlight on a real device, so whether they can be found is unverified. Commands for checking your own site:
# Does robots.txt mention Applebot or Googlebot?
curl -s https://example.com/robots.txt | grep -i -B1 -A3 -E "applebot|googlebot"
# Does the page answer the Applebot User-Agent?
curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)" https://example.com/page/
To confirm the real Applebot has visited, check your server logs. Apple's page gives two ways to verify it, reverse DNS (*.applebot.apple.com) and a published IP range; for how to look in the logs, see How do I check whether GPTBot and other AI crawlers have visited my site?.
Where do Siri's web answers come from?
Apple has said only half of it. Its page says data fetched by Applebot may supply background and up-to-date content for AI model output, for example when Siri and Search answer broad world-knowledge questions, possibly with links to the sources used. Which pages get picked, which search service is used and how results are ranked are not stated on that page, and we have not verified them.
Another path is ChatGPT. Apple's "Turn on ChatGPT on iPhone" page says that once you turn on the ChatGPT extension under Settings > Siri, Apple Intelligence can work with OpenAI's ChatGPT; you can choose to confirm before sending, and the request is sent to ChatGPT to answer. On this path the recommendation rules are ChatGPT's own, see How do I get ChatGPT to recommend my product?. The page also says Apple Intelligence is not available in mainland China; check Apple's page for regions and languages.
Apple's page lists five factors that search ranking may consider, with no predetermined weights:
| Factor listed by Apple | What you can do |
|---|---|
| Aggregated user engagement with search results | Write clear titles and descriptions so people who click get their answer |
| Relevance of search terms to page topics and content | One topic per page, with the body answering the question directly |
| Number and quality of links from other pages | Get mentioned in real articles, directories and communities; see Do brand mentions matter more than backlinks for AI search? What two studies say |
| Approximate user location | Nothing to do; for multilingual sites make sure every language version can be crawled |
| Page design characteristics | Apple does not elaborate, and we have not verified what this means |
How do you check whether your site is visible to Apple's search?
1. Confirm Applebot is not blocked
Run the curl commands above and confirm no Disallow for Applebot or Googlebot in robots.txt covers your content pages.
2. Confirm the page has no noindex
Look at the robots meta in the page head and the X-Robots-Tag in the response headers. Apple's page says pages with noindex do not appear in Spotlight or Siri suggestions.
3. Put the body text directly in the HTML
The reason is in Do AI crawlers render JavaScript?.
4. Search a few times on an iPhone or Mac
Ask Siri the buyer's questions and search for your brand name in Spotlight, and note whether you appear. Apple Intelligence is limited to certain regions and languages, so check Apple's page; for how to record results, see How do you measure AI visibility? Four metrics and a table you can follow.
5. Check whether Applebot has appeared in the logs
Search the logs for Applebot; commands are in How do I check whether GPTBot and other AI crawlers have visited my site?.
FAQ
What is Applebot-Extended's User-Agent?
It has no crawling identity of its own. Apple's page says Applebot-Extended does not crawl pages and is used only to decide whether data fetched by Applebot may be used for training. So in your server logs look for Applebot, not Applebot-Extended.
If you block Applebot-Extended, can Siri and Spotlight still find you?
Yes. Apple's page says pages that disallow Applebot-Extended can still appear in search results, and its rules are not considered in Search ranking; even with nosnippet added as well, content remains discoverable through Spotlight, Siri and Safari.
How do I keep a page out of Siri's answers?
Apple's page says to add nosnippet to the specific content (a meta tag or X-Robots-Tag). It also states the cost: Applebot no longer generates a description or web answer for the page, and suggestions show only the title. Do not add it to pages you want recommended.
Which search service do Siri's web answers use?
Apple's page does not say, and we have not verified it. All we can confirm is two points it makes: Applebot data supports search in products such as Siri, and may also supply background and source links when answering broad world-knowledge questions.
Can I submit a URL to Apple?
Apple's Applebot page does not describe a way to submit URLs; it only gives a contact email, applebot@apple.com. What you can do is make the site crawlable for Applebot and keep normal internal links and a sitemap.
Sources
- Apple · About Applebot (Viewed 2026-11-09; the page shows a publication date of 2026-09-04)
- Apple · Turn on ChatGPT on iPhone (checked 2026-11-09)
- kagentx · kagentx.com/robots.txt (Read 2026-11-09)
- MochiStamp · mochistamp.com/robots.txt (Read 2026-11-09)
- Gamma AI Slide · gammaslide.com/robots.txt (Read 2026-11-09)
- Pagent · pagent.kagentx.com/robots.txt (Read 2026-11-09)
- kagentx evaluations: the data on the case study pages; each example gives its date.