Guide · Crawler logs

How do I check whether GPTBot and other AI crawlers have visited my site?

Updated 2026-11-07 · kagentx

Short answer

Search the access log for the crawler name, then check the source against the official IP list. In nginx and Apache combined logs, pull out the User-Agent with awk and grep for GPTBot, ClaudeBot and PerplexityBot to count requests; a User-Agent can be forged, so compare the IP with the JSON ranges published by OpenAI, Anthropic and Perplexity. With Cloudflare, look directly in the Crawlers tab of AI Crawl Control.

How do I see in the logs whether GPTBot has visited?

Search for the User-Agent. In the default combined log format of nginx and Apache, the User-Agent is the last quoted field; split on quotes with awk, take field 6, then count which crawler names appear. We tested the commands below on this machine with a fake log:

# Which AI bots hit the site, and how often (nginx or Apache "combined" log format)
awk -F'"' '{print $6}' /var/log/nginx/access.log \
  | grep -o -E 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User' \
  | sort | uniq -c | sort -rn

The output is the request count for each crawler, one per line. Apache differs only in the log path:

# Apache: same command, different path (Debian/Ubuntu: /var/log/apache2/access.log, RHEL: /var/log/httpd/access_log)
awk -F'"' '{print $6}' /var/log/apache2/access.log \
  | grep -o -E 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User' \
  | sort | uniq -c | sort -rn

Once you see a crawler, three questions usually follow: what status code it got, which pages it asked for, and whether the source IP is genuine.

# What did GPTBot get back? (status code in column 9)
grep GPTBot /var/log/nginx/access.log | awk '{print $9}' | sort | uniq -c
# Which pages did GPTBot request?
grep GPTBot /var/log/nginx/access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Status 200 means it read the page; 403 usually means a firewall or server rule blocked it; 404 means the address it asked for does not exist. When logs have been rotated and compressed to .gz, search them together with zgrep:

# Rotated logs (.gz) as well
zgrep -h GPTBot /var/log/nginx/access.log*.gz | awk '{print $1}' | sort | uniq -c | sort -rn | head

How do I confirm a request really came from the official crawler?

Check the source IP against the official list. The User-Agent is filled in by the request itself and anyone can write GPTBot; the official crawlers send from published IP ranges. All three publish JSON lists; the table shows what we read when we opened them on 2026-11-07:

CrawlerOfficial IP listcreationTime when we read it
GPTBothttps://openai.com/gptbot.json2026-09-22
OAI-SearchBothttps://openai.com/searchbot.json2026-01-02
ChatGPT-Userhttps://openai.com/chatgpt-user.json2026-11-06
ClaudeBot, Claude-SearchBot, Claude-Userhttps://claude.com/crawling/bots.json2026-10-20
PerplexityBothttps://www.perplexity.ai/perplexitybot.json2026-10-28
Perplexity-Userhttps://www.perplexity.ai/perplexity-user.json2026-10-28

User-Agent examples on the official pages (checked 2026-11-07):

CrawlerUser-Agent given on the official pageSources
GPTBotMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbotdevelopers.openai.com/api/docs/bots
OAI-SearchBotMozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbotdevelopers.openai.com/api/docs/bots
ChatGPT-UserMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/botdevelopers.openai.com/api/docs/bots
PerplexityBotMozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)docs.perplexity.ai/guides/bots
Perplexity-UserMozilla/5.0 (compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)docs.perplexity.ai/guides/bots

Both OpenAI’s and Perplexity’s pages say the version numbers change, so search the logs for the name (GPTBot), not the whole string. For Anthropic’s help page we verified only the purpose of the three names and the IP list address, not the full User-Agent strings, so we do not list them here. It also warns that blocking by IP may not be reliable and recommends managing with robots.txt. For a side-by-side of every crawler see AI crawler user agents: a side-by-side table.

To check: first save the de-duplicated source IPs for one crawler to a file, then compare them with the official ranges using a short piece of Python.

grep GPTBot /var/log/nginx/access.log | awk '{print $1}' | sort -u > ips.txt
# check_ip.py: is each IP on the bot's official list?  usage: python3 check_ip.py <list url> < ips.txt
import ipaddress, json, sys, urllib.request
req = urllib.request.Request(sys.argv[1], headers={"User-Agent": "ip-check"})
nets = [ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix"))
        for p in json.load(urllib.request.urlopen(req))["prefixes"]]
for line in sys.stdin:
    ip = line.strip()
    if ip:
        print(ip, "on list" if any(ipaddress.ip_address(ip) in n for n in nets) else "NOT on list")
python3 check_ip.py https://openai.com/gptbot.json < ips.txt

In the output, “on list” means an address inside the official ranges, and “NOT on list” means it is not the official crawler, so you can handle it in your firewall as you see fit. The lists are updated, so download them fresh each time and do not keep old ranges for long-term use.

If you use Cloudflare, where do I look?

Open AI Crawl Control. Cloudflare’s documentation (checked 2026-11-07) gives this route: log in to the dashboard, select your account and domain, and open AI Crawl Control. The Crawlers tab lists each AI crawler with its operator, request count, data transfer and current action; the Metrics tab has requests over time, the status code distribution (2xx, 3xx, 4xx, 5xx) and the most requested paths, can be filtered by crawler, operator, hostname and path, and can export CSV. The documentation says the feature is available on all plans, while referral analytics are for paid plans only.

These numbers need not match what the server log shows: Cloudflare sees requests that pass through it, the origin log sees requests that reach the origin, and anything served from cache never reaches the origin log. For whether it blocks by default and how to allow crawlers see Does Cloudflare block AI crawlers by default? How to check and allow them. When the site sits behind a CDN, the source IP in the origin log may be the CDN’s address; to do the IP check above you need the real client IP field that the CDN passes along.

I found no record. Why?

  1. 1. Check the log range first

    If logs are kept for only a few days, not finding anything does not mean it never came. Search rotated .gz files with zgrep.

  2. 2. Confirm robots.txt is not blocking it

    Use AI crawler check to see whether each crawler is “allowed”; for how to write it see How to write robots.txt for AI crawlers.

  3. 3. Confirm the firewall is not blocking it

    A 403 in the log means it arrived but was blocked. For Cloudflare settings see Does Cloudflare block AI crawlers by default? How to check and allow them.

  4. 4. Confirm the body text is served directly

    If it reads the page but the body is empty, it will not be cited either; see Do AI crawlers render JavaScript?.

  5. 5. A crawler having visited does not mean you are recommended

    A crawler having visited only means it read the page. Whether you appear in answers is something you have to ask and read the sources for; see How do you measure AI visibility? Four metrics and a table you can follow. For visits that AI sends you, see How to see ChatGPT traffic in GA4.

What to do for each assistant

Once you have confirmed the crawlers visited, each assistant has its own approach: How to get ChatGPT to recommend my product, How to get Perplexity to recommend my product, How to get Meta AI to recommend my product.

FAQ

How do I know whether GPTBot has crawled my site?

Search the server access log for GPTBot. In the default nginx and Apache log formats the User-Agent is field 6 when split on quotes, so grep GPTBot shows the requests. Then compare the source IP with OpenAI’s official gptbot.json to confirm it is genuine.

What does it mean if there is no GPTBot at all in the log?

It only means the stretch of log you searched has none. The crawler may not have come yet, the logs may have been rotated and compressed (search .gz files with zgrep), or the site may sit behind a CDN so that the origin log shows the CDN’s address. Being disallowed in robots.txt would also keep it away.

Can a User-Agent be forged?

Yes. Anyone can label their requests as GPTBot. So check the source address against the officially published IP list; anything not on the list is not the official crawler.

What is the difference between GPTBot, OAI-SearchBot and ChatGPT-User in the logs?

OpenAI’s official page separates them into three: GPTBot is for training, OAI-SearchBot is for ChatGPT search, and ChatGPT-User comes when a user asks ChatGPT to open a web page. Each has its own IP list.

If I use Cloudflare, where do I look?

The official documentation says: log in to the Cloudflare dashboard, select your account and domain, and open AI Crawl Control. The Crawlers tab shows each AI crawler’s requests and data transfer, and the Metrics tab shows the status code distribution and the most requested paths. The documentation says the feature is available on all plans, with referral analytics for paid plans only.

Sources

kagentx

Can crawlers get in?
Let the evaluation check

The evaluation checks whether crawlers can get in and whether the text is served directly, then asks ChatGPT, Perplexity and Claude (with web search on) your buyers' questions and sees whether you are mentioned. The Basic plan is $1, and results usually take 2–3 minutes.