Find AI bots in your server logs (and check they're real)

Tools like Google Analytics count visits with a small script that runs in the browser. A crawler that only downloads your HTML never runs that script, so it never shows up.

Your web server writes down every request, from people and bots alike. That record is your access log.

Logs answer three questions: which bots visit, which pages they fetch, and whether they get the page or an error.

• nginx: the standard “combined” format already has everything you need. On Debian and Ubuntu the file is usually /var/log/nginx/access.log.
• Apache: use the combined format too. The “common” format has no user agent, so you can't spot bots with it. Add %{User-agent}i to your LogFormat.
• Behind a CDN or proxy? Then the first field may show your proxy's IP for every request. Set up your server's real-IP setting (or log the forwarded client IP) first, or every IP check below will fail.
• No access to raw logs? Look in your host's dashboard, or put a CDN in front of your site. More on that below.

Here's one line in the combined format, with a made-up IP address. Each part answers a question.

203.0.113.7 - - [19/Oct/2026:08:01:02 +0000] "GET /pricing HTTP/1.1" 200 5123 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot"

• IP address: where the request came from. You'll check this against the official lists.
• Time: when it happened.
• Request: the method and the page. Here GET /pricing.
• Status code: 200 means served, 301/302 redirected, 403 blocked, 404 missing, 5xx a server error.
• Bytes: how much was sent back.
• User agent: who the visitor says it is. Here GPTBot. Anyone can type this, so it proves nothing on its own.

This list comes from each company's own documentation, checked on 19 October 2026.

Google-Extended and Applebot-Extended are robots.txt tokens, not crawlers. They never appear in a log.

Anthropic doesn't publish full user agent strings. In logs they look like “compatible; Claude-User/1.0; +claudebot@anthropic.com”.

These work on a combined-format log. We tested them on 19 October 2026 with macOS grep and awk. Replace access.log with your file.

grep -oE 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User|meta-externalagent|Amazonbot|Applebot|DuckAssistBot|MistralAI-User|CCBot' access.log | sort | uniq -c | sort -rn

awk -F'"' '{ split($3,a," "); if (match($6,/GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User/)) print substr($6,RSTART,RLENGTH), a[1] }' access.log | sort | uniq -c

grep 'OAI-SearchBot' access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head

awk -F'"' '$6 ~ /GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User/ { split($3,a," "); if (a[1] >= 400) print a[1], $2 }' access.log

zgrep -c 'GPTBot' /var/log/nginx/access.log.*.gz

Check every bot IP with Python

This script downloads the official IP lists and prints VERIFIED or NOT IN LIST for each bot line. It only uses Python's standard library.

# Usage: python3 verify_bots.py access.log
import ipaddress, json, re, sys, urllib.request

LISTS = {
    "GPTBot": "https://openai.com/gptbot.json",
    "OAI-SearchBot": "https://openai.com/searchbot.json",
    "ChatGPT-User": "https://openai.com/chatgpt-user.json",
    "ClaudeBot": "https://claude.com/crawling/bots.json",
    "Claude-User": "https://claude.com/crawling/bots.json",
    "Claude-SearchBot": "https://claude.com/crawling/bots.json",
    "PerplexityBot": "https://www.perplexity.com/perplexitybot.json",
    "Perplexity-User": "https://www.perplexity.com/perplexity-user.json",
    "Googlebot": "https://developers.google.com/static/crawling/ipranges/common-crawlers.json",
    "bingbot": "https://www.bing.com/toolbox/bingbot.json",
}
cache = {}

def networks(url):
    if url not in cache:
        req = urllib.request.Request(url, headers={"User-Agent": "log-check/1.0"})
        data = json.load(urllib.request.urlopen(req, timeout=20))
        cache[url] = [ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix"))
                      for p in data["prefixes"]]
    return cache[url]

bot = re.compile("|".join(sorted(map(re.escape, LISTS), key=len, reverse=True)))
for line in open(sys.argv[1], errors="replace"):
    ua = line.split('"')[5] if line.count('"') >= 6 else line
    match = bot.search(ua)
    if not match:
        continue
    try:
        ip = ipaddress.ip_address(line.split()[0])
    except ValueError:
        continue
    ok = any(ip in net for net in networks(LISTS[match.group(0)]))
    print("VERIFIED   " if ok else "NOT IN LIST", match.group(0), ip)

A user agent is just text. Anyone can send “GPTBot”. Perplexity itself recommends combining user agent matching with IP verification.

Way 1: the official IP lists

OpenAI, Anthropic, Perplexity, Google, Microsoft (Bing), Apple, DuckDuckGo and Common Crawl publish JSON lists of the IP ranges their bots use. If the IP is in the list, the request came from that company's network. The links are in the table above.

Way 2: reverse DNS

Google: look up the IP's hostname. It must end in googlebot.com, google.com or googleusercontent.com, and looking up that name must give the same IP back. Bing works the same way, with names ending in search.msn.com.

host 66.249.66.1
# 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
host crawl-66-249-66-1.googlebot.com
# crawl-66-249-66-1.googlebot.com has address 66.249.66.1  ← same IP: real Googlebot

• Training, search and user requests are different. GPTBot and ClaudeBot collect training data. OAI-SearchBot, Claude-SearchBot and PerplexityBot build search indexes. ChatGPT-User, Claude-User and Perplexity-User fetch a page because a person asked about it right now.
• Lots of 403 or 5xx answers mean you're blocking or breaking AI access. Check your firewall, your CDN's bot settings and your robots.txt.
• robots.txt and llms.txt requests show which bots read your rules. OpenAI says OAI-SearchBot and GPTBot may add a robots.txt marker to their user agent when they fetch that file.
• User requests may skip robots.txt. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them.
• A visit isn't a citation. It means a bot read your page, not that any AI answer will quote it.

• Set up robots.txt for AI crawlers
• How to get your pages cited by AI

Cloudflare's AI Crawl Control (formerly AI Audit) shows which AI services crawl your site and which crawlers ignore your rules. Cloudflare says it's available on all plans.

Cloudflare Logpush can export request logs with fields such as ClientIP, ClientRequestUserAgent, ClientRequestPath and EdgeResponseStatus. Our analyzer reads those JSON lines too.

IP addresses can be personal data. The EU Court of Justice said so for dynamic IP addresses in the Breyer case (C-582/14, 19 October 2016), and recital 30 of the GDPR names IP addresses as online identifiers.

Bot IPs belong to companies, but the same log also holds your visitors' IPs. A practical approach:

• Filter out the bot lines first, and work with those.
• Keep raw logs only as long as you need them, for example for security. The GDPR sets no fixed number of days; it says not to keep personal data longer than necessary.
• Store totals (bot, day, page, status, verified or not) instead of IPs.
• Shorten or hash an IP once you've checked it.
• Don't upload raw logs to an outside tool without a data processing agreement. Our analyzer runs in your browser, so nothing is uploaded.

This is general information, not legal advice.

• Free AI Bot Log Analyzer (runs in your browser)
• Configure robots.txt for AI crawlers
• Reference: AI bot user agents
• The llms.txt guide
• Robots.txt generator for AI bots
• Track visits from AI tools