AI crawlers
ClaudeBot
Also called: Claude-User
ClaudeBot is Anthropic’s web crawler that collects content which could contribute to training its generative AI models.
In plain words
Anthropic runs three bots. ClaudeBot is for training, Claude-User fetches pages when a person asks Claude something, and Claude-SearchBot works on search quality.
If a site restricts ClaudeBot, Anthropic says that signals the site’s future content should be left out of its training datasets.
Example
The snippet below is Anthropic’s own example for blocking ClaudeBot from a whole site. Anthropic adds that you need to do this for every subdomain you want to opt out.
User-agent: ClaudeBot
Disallow: /Why it matters for AI search
Blocking ClaudeBot is about training. Anthropic says disabling Claude-User or Claude-SearchBot may reduce your site’s visibility in search.
Anthropic also says blocking its IP addresses may not work reliably, because it stops the bots from reading your robots.txt.
What you can do
- Use robots.txt rather than IP blocking to opt out, as Anthropic recommends.
- Decide separately about ClaudeBot (training) and Claude-SearchBot (search).
- If its bots visit too often, add a Crawl-delay line; Anthropic says it supports this non-standard extension.
Try it: Free robots.txt generator
Often confused with
- Claude-User
- Claude-User visits a page when a person asks Claude a question. Blocking it stops Claude from fetching your content for those users.
- Claude-SearchBot
- Claude-SearchBot indexes content to improve Claude’s search results.
Quick facts
- Run by
- Anthropic
- robots.txt name
- ClaudeBot, Claude-User
- Official IP list
- crawling/bots.json
Sources
Facts and sources last checked on 19 October 2026.
Related terms
How does your page do in AI search?
GEO-Score checks one page on 22 signals, from AI crawler access to structured data, and shows what to fix first.
Analyze a page