Configuring robots.txt for AI Crawlers
A practical walkthrough for GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Google-Extended and the rest: what each one does, and a robots.txt you can actually use.
Most sites either block AI crawlers by accident, through an inherited WAF rule or a copy-pasted robots.txt template, or allow everything without thinking about what each bot is actually for. This guide is the fix for both.
Check My Crawler AccessTwo Kinds of AI Crawler, Not One
AI companies run at least two different kinds of crawler, and treating them as one thing is the most common robots.txt mistake. Training crawlers, such as GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google) and CCBot (Common Crawl, used by many models indirectly), visit on their own schedule to gather data for a future model version. Blocking one does not remove anything already learned from a past crawl; it only opts you out of the next one.
Real-time fetch crawlers work differently: OAI-SearchBot and ChatGPT-User (OpenAI), PerplexityBot (Perplexity), and the search-connected fetches Claude and Gemini make, all run at the moment a user asks a question that needs your page right now. Blocking these has an immediate effect: the engine cannot read your page for that answer, full stop, even if your content is exactly what it needed.
This is why a blanket 'block all bots' rule is usually a mistake, and so is a blanket 'allow everything' rule with no thought behind it. The right robots.txt treats each crawler by what it does: allow the ones that drive citations and real-time answers, and make a deliberate, informed choice, not a default, about the training-only ones.
What to Decide Before You Edit robots.txt
Know Which Type You're Blocking
Before disallowing anything, check whether it is a training crawler, which affects future model versions only, or a real-time fetcher, which affects whether you are cited in an answer right now. The two have completely different consequences.
Verify the Bot Is Real
A user-agent string is just text; anyone can send 'GPTBot' in a request header. For anything beyond a basic robots.txt rule, such as a WAF block, verify by published IP ranges or a CDN's verified-bot list, not the string alone.
Check Your Server Logs, Not Just robots.txt
robots.txt is a request, not a lock. Well-behaved bots respect it, but the only way to know your rules are actually working is to sample server logs for 200 responses, not silent drops or WAF challenges, from the bots you meant to allow.
Why This Is Worth Getting Right
Accidental Blocking Is the Default, Not the Exception
Independent research from Originality.ai found GPTBot blocked by over a third of the top 1,000 websites, and interviews with site owners showed many of those blocks were inherited from default WAF rules or copy-pasted templates, not a deliberate decision.
Usually a Five-Minute Fix
Once you know which bots you want to allow, the actual robots.txt change is a handful of lines. The hard part is knowing what to write, not writing it.
Blocking Has a Measured Cost
Case studies analyzed by Hack/Hackers found sites that blocked AI crawlers subsequently saw a meaningful drop in referral traffic from AI answer engines. The crawler you block today is the citation you do not get tomorrow.
Set Up AI Crawler Access in 3 Steps
Decide Per Bot, Not All-Or-Nothing
List which crawlers you want to allow. Most sites should allow the real-time fetchers, such as OAI-SearchBot, ChatGPT-User and PerplexityBot, by default, since blocking them removes you from live answers immediately.
Write Explicit Allow Rules
Add a User-agent block per bot rather than relying on a generic wildcard rule inherited from a template. Explicit rules override defaults and make your intent auditable later.
Confirm With Server Logs
After deploying, sample your logs for the bots you allowed. Look for 200 responses, not 403s or WAF challenges. A bot that respects robots.txt but still gets blocked elsewhere in your stack needs a separate fix.
AI Crawler Access FAQ
What's the difference between a training crawler and a real-time fetch crawler?
A training crawler, such as GPTBot, ClaudeBot or Google-Extended, gathers data on its own schedule to improve a future model version — blocking it has no effect on citations happening today. A real-time fetch crawler, such as OAI-SearchBot, ChatGPT-User or PerplexityBot, runs at the moment a user's question needs your page — blocking it means that specific answer cannot use your content at all.
What does a working robots.txt for AI crawlers look like?
A minimal example that allows the main citation-relevant bots: User-agent: GPTBot, Allow: /; User-agent: OAI-SearchBot, Allow: /; User-agent: ChatGPT-User, Allow: /; User-agent: ClaudeBot, Allow: /; User-agent: PerplexityBot, Allow: /; User-agent: Google-Extended, Allow: /; then a Sitemap line pointing to your sitemap.xml. Adjust which bots get Allow versus Disallow based on your own training-data decision.
Should I block GPTBot to keep AI out of my content?
For most sites, no. GPTBot is OpenAI's training crawler — blocking it stops future models from training on new content, but does not remove anything already learned, and does not affect ChatGPT's real-time citations, which come from OAI-SearchBot and ChatGPT-User instead. Blocking it is a deliberate opt-out decision, not a security measure, and case-study data suggests it can cost real referral traffic without a clear upside for most publishers.
Does blocking AI crawlers affect my Google ranking?
Blocking GPTBot, ClaudeBot or PerplexityBot has no direct effect on Google Search ranking; Googlebot is a separate crawler entirely. Blocking Google-Extended specifically is different: it stops your content from being used in Google's own AI features, such as AI Overviews and Gemini, without touching your regular search ranking, since that's governed by Googlebot.
How do I know a request claiming to be GPTBot is actually GPTBot?
Never trust the user-agent string alone; it is trivial to fake. For anything beyond a basic robots.txt rule, verify against the crawler's published IP ranges (OpenAI, Anthropic and others publish these) or use a CDN's verified-bot list, such as Cloudflare's AI Crawl Control, which checks more than just the header.
Do I also need an llms.txt file?
llms.txt is an emerging, unofficial convention some sites use to point AI crawlers at a curated summary of their content. It is not a standard any major AI company has confirmed it reads, and it does not replace robots.txt, which is the actual access-control mechanism crawlers respect. Worth watching, not yet worth prioritizing over getting your robots.txt right.
Check Whether AI Crawlers Can Actually Read Your Site
GEO-Score inspects your robots.txt, server responses and JavaScript-rendered content to show exactly which AI bots are allowed in. Five page checks per domain are free every 30 days.
Check My Crawler Access