Which crawlers does Meta have, and what does each do?
Five, of which meta-webindexer and meta-externalfetcher relate most directly to Meta AI answers. The table below is from Meta’s official developer crawler page (opened and checked 2026-11-07; the page carries no date):
| User-Agent | What the docs say it is for | robots.txt |
|---|---|---|
| meta-webindexer/1.1 | Analyzes web content to improve the quality of Meta AI search results; allowing it helps Meta cite and link to your content in answers | The page does not say whether it bypasses it |
| meta-externalfetcher/1.1 | Fetches a link when a user requests it, supporting agentic features (helping AI complete tasks on websites) | The page says it may bypass robots.txt |
| meta-externalagent/1.1 | Crawls the web to train foundation AI models, or to index content directly to improve products | The page does not say whether it bypasses it |
| meta-externalads/1.1 | Crawls the web for advertising and other business products and services | The page does not say whether it bypasses it |
| facebookexternalhit/1.1 | Fetches links shared on Facebook, Instagram and Messenger to generate the title, description and thumbnail | The page says it may bypass it during security or integrity checks |
The official page gives no IP list or ASN, only saying that allowing by IP is more reliable than by User-Agent. A robots.txt change can take up to 24 hours to take effect. For a side-by-side of every crawler see AI crawler user agents: a side-by-side table.
Which search engine do Meta AI's web results come from?
A 2024 report says Google and Bing. In an article dated 2024-04-18, Search Engine Journal reported that Meta announced its AI assistant (then powered by Llama 3) would add real-time search results from Google and Bing, choosing one of them per query; the author’s own test only triggered Bing and could not tell how it chooses. We did not open Meta’s original announcement, this is a 2024 account, whether it has changed since is unverified, and Meta’s official documentation does not say how sources are ranked.
The practical meaning: whatever Meta’s own crawlers do, your pages first need to be in the Google and Bing indexes. See How to use Google Search Console on a new site and How to use Bing Webmaster Tools.
How should robots.txt be written so Meta AI can read your site?
When there is no rule aimed at them and no site-wide block, you need not write anything. To spell it out, allow the two that relate to answers.
User-agent: meta-webindexer Allow: / User-agent: meta-externalfetcher Allow: /
meta-externalagent is a different matter: the official page says it is used to train foundation models or build an index. Whether to allow it depends on your view of training use; the trade-offs are in How to write robots.txt for AI crawlers. Do not, in trying to block training use, write a site-wide block for every name starting with meta-, or meta-webindexer cannot get in either.
How do you confirm Meta's crawlers actually visited?
1. Count them in the server log
The command below counts requests from each Meta crawler in an nginx log. For more commands and how to verify IPs see How to check your server logs for AI crawlers.
# Count requests from Meta crawlers in an nginx access log grep -i -E "meta-webindexer|meta-externalagent|meta-externalfetcher|facebookexternalhit" /var/log/nginx/access.log \ | awk -F'"' '{print $6}' | grep -o -i -E "meta-[a-z]+|facebookexternalhit" | sort | uniq -c | sort -rn2. Check whether a firewall blocks it separately
A correct robots.txt does not mean it can get in. A firewall or CDN bot block can stop it outside robots.txt, especially meta-externalfetcher, because it may not follow robots.txt.
3. Do not identify a crawler by User-Agent alone
A User-Agent can be forged. Meta’s official page gives no IP list, so the numbers in the log only show that requests claimed to be these crawlers.
How should pages be written to be more likely cited by Meta AI?
The official documentation gives no ranking rules, so the following are basics that hold for all AI search:
| What to do | Why | Which guide |
|---|---|---|
| Get your pages indexed by Google and Bing first | The 2024 report says its web results come from these two | See the Search Console and Bing Webmaster Tools guides |
| Body text directly in HTML, with a direct answer at the start | Crawlers read the page at request time; content that appears only after scripts run is missed | See How to write content AI will quote |
| Link social-platform content back to your own site | Put what you want cited on pages you control | See Do AI answers cite posts on X and LinkedIn? |
Related guides: How to write content AI will quote: before and after, Do AI answers cite posts on X and LinkedIn?, How do you measure AI visibility? Four metrics and a table you can follow.
How do you know whether Meta AI mentions you?
1. Write down 5 questions buyers ask
Leave out your brand name and write it the way a buyer would.
2. Ask each question three times in Meta AI
Note whether it mentions you, whom it recommends, and which links it gives. How Meta AI is accessed and where it is available varies, which we have not verified; test through whichever entry point you can use.
3. Test again with the same questions two to four weeks later
For how to record it see How do you measure AI visibility? Four metrics and a table you can follow.
How is it different from other AI assistants?
What is distinctive about Meta is how finely its crawlers are split: one for search, one for training, and one that only comes when a user asks, each under a different name. For other assistants see How to get ChatGPT to recommend my product, How to get Google Gemini to recommend my product, How to get Microsoft Copilot to recommend my product, How to get Grok to recommend my product.
FAQ
Which search engine does Meta AI use?
A report dated 2024-04-18 says Meta announced that its AI assistant would add real-time search results from Google and Bing, choosing one of them per query. That is a 2024 account, and whether it has changed since is unverified.
What is meta-webindexer for?
Meta’s official crawler page says: it analyzes web content to improve the quality of Meta AI search results, and allowing it helps Meta cite and link to your content in answers.
Will meta-externalagent use my content to train models?
The official page says it crawls the web to train foundation AI models, or to index content directly to improve products. To block training use, add a Disallow rule for it in robots.txt as the official page describes; it is a different name from meta-webindexer, which serves Meta AI search results.
Does meta-externalfetcher obey robots.txt?
The official page says it may bypass robots.txt. It only comes when a user asks Meta AI to open a link, so for those requests to read your page, what matters is not blocking it in the firewall.
Has Meta published an IP list for its crawlers?
The official page we read gives no IP list or ASN, only saying that allowing by IP is more reliable than by User-Agent. Where an IP list is published we have not verified.
Sources
- Meta · Meta Web Crawlers (Opened and checked 2026-11-07; the page carries no date)
- Search Engine Journal · Meta integrates Google and Bing search results into its AI assistant (Published 2024-04-18, read 2026-11-07)
- kagentx evaluations: the data on the case study pages; each example gives its date.