A practical walkthrough for GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Google-Extended and the rest: what each one does, and a robots.txt you can actually use.
Most sites either block AI crawlers by accident, through an inherited WAF rule or a copy-pasted robots.txt template, or allow everything without thinking about what each bot is actually for. This guide is the fix for both.
Check My Crawler AccessAI companies run at least two different kinds of crawler, and treating them as one thing is the most common robots.txt mistake. Training crawlers, such as GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google) and CCBot (Common Crawl, used by many models indirectly), visit on their own schedule to gather data for a future model version. Blocking one does not remove anything already learned from a past crawl; it only opts you out of the next one.
Real-time fetch crawlers work differently: OAI-SearchBot and ChatGPT-User (OpenAI), PerplexityBot (Perplexity), and the search-connected fetches Claude and Gemini make, all run at the moment a user asks a question that needs your page right now. Blocking these has an immediate effect: the engine cannot read your page for that answer, full stop, even if your content is exactly what it needed.
This is why a blanket 'block all bots' rule is usually a mistake, and so is a blanket 'allow everything' rule with no thought behind it. The right robots.txt treats each crawler by what it does: allow the ones that drive citations and real-time answers, and make a deliberate, informed choice, not a default, about the training-only ones.
Before disallowing anything, check whether it is a training crawler, which affects future model versions only, or a real-time fetcher, which affects whether you are cited in an answer right now. The two have completely different consequences.
A user-agent string is just text; anyone can send 'GPTBot' in a request header. For anything beyond a basic robots.txt rule, such as a WAF block, verify by published IP ranges or a CDN's verified-bot list, not the string alone.
robots.txt is a request, not a lock. Well-behaved bots respect it, but the only way to know your rules are actually working is to sample server logs for 200 responses, not silent drops or WAF challenges, from the bots you meant to allow.
Independent research from Originality.ai found GPTBot blocked by over a third of the top 1,000 websites, and interviews with site owners showed many of those blocks were inherited from default WAF rules or copy-pasted templates, not a deliberate decision.
Once you know which bots you want to allow, the actual robots.txt change is a handful of lines. The hard part is knowing what to write, not writing it.
Case studies analyzed by Hack/Hackers found sites that blocked AI crawlers subsequently saw a meaningful drop in referral traffic from AI answer engines. The crawler you block today is the citation you do not get tomorrow.
List which crawlers you want to allow. Most sites should allow the real-time fetchers, such as OAI-SearchBot, ChatGPT-User and PerplexityBot, by default, since blocking them removes you from live answers immediately.
Add a User-agent block per bot rather than relying on a generic wildcard rule inherited from a template. Explicit rules override defaults and make your intent auditable later.
After deploying, sample your logs for the bots you allowed. Look for 200 responses, not 403s or WAF challenges. A bot that respects robots.txt but still gets blocked elsewhere in your stack needs a separate fix.
A training crawler, such as GPTBot, ClaudeBot or Google-Extended, gathers data on its own schedule to improve a future model version — blocking it has no effect on citations happening today. A real-time fetch crawler, such as OAI-SearchBot, ChatGPT-User or PerplexityBot, runs at the moment a user's question needs your page — blocking it means that specific answer cannot use your content at all.
A minimal example that allows the main citation-relevant bots: User-agent: GPTBot, Allow: /; User-agent: OAI-SearchBot, Allow: /; User-agent: ChatGPT-User, Allow: /; User-agent: ClaudeBot, Allow: /; User-agent: PerplexityBot, Allow: /; User-agent: Google-Extended, Allow: /; then a Sitemap line pointing to your sitemap.xml. Adjust which bots get Allow versus Disallow based on your own training-data decision.
For most sites, no. GPTBot is OpenAI's training crawler — blocking it stops future models from training on new content, but does not remove anything already learned, and does not affect ChatGPT's real-time citations, which come from OAI-SearchBot and ChatGPT-User instead. Blocking it is a deliberate opt-out decision, not a security measure, and case-study data suggests it can cost real referral traffic without a clear upside for most publishers.
Blocking GPTBot, ClaudeBot or PerplexityBot has no direct effect on Google Search ranking; Googlebot is a separate crawler entirely. Blocking Google-Extended specifically is different: it stops your content from being used in Google's own AI features, such as AI Overviews and Gemini, without touching your regular search ranking, since that's governed by Googlebot.
Never trust the user-agent string alone; it is trivial to fake. For anything beyond a basic robots.txt rule, verify against the crawler's published IP ranges (OpenAI, Anthropic and others publish these) or use a CDN's verified-bot list, such as Cloudflare's AI Crawl Control, which checks more than just the header.
llms.txt is an emerging, unofficial convention some sites use to point AI crawlers at a curated summary of their content. It is not a standard any major AI company has confirmed it reads, and it does not replace robots.txt, which is the actual access-control mechanism crawlers respect. Worth watching, not yet worth prioritizing over getting your robots.txt right.
GEO-Score inspects your robots.txt, server responses and JavaScript-rendered content to show exactly which AI bots are allowed in. Five page checks per domain are free every 30 days.
Check My Crawler Access