Complete reference for robots.txt configuration
A user-agent is a string that identifies which bot or browser is accessing your website. AI search engines use specific user-agents so you can control their access in robots.txt.
Understanding these user-agents helps you configure GEO-friendly access. This reference lists all major AI bots you should know about.
| Bot Name | User-Agent String | Company |
|---|---|---|
| GPTBot | GPTBot |
| OpenAI (ChatGPT) |
| ChatGPT-User | ChatGPT-User | OpenAI (ChatGPT Plugins/Browse) |
| Claude-Web | Claude-Web | Anthropic (Claude) |
| anthropic-ai | anthropic-ai | Anthropic (Training) |
| Google-Extended | Google-Extended | Google (Gemini) |
| PerplexityBot | PerplexityBot | Perplexity AI |
| YouBot | YouBot | You.com |
| Applebot-Extended | Applebot-Extended | Apple (AI features) |
| Diffbot | Diffbot | Diffbot (Knowledge Graph) |
| cohere-ai | cohere-ai | Cohere |
Visit:
Verify the file loads and contains your AI bot configurations.
Google Search Console offers a robots.txt tester. Test different user-agents to verify your configuration.
Check your server logs to see which bots are actually crawling. Look for the user-agent strings in access logs.
Allow all AI bots unless you have specific reasons not to
Keep your robots.txt simple and well-documented
Test changes before deploying to production
Monitor logs to see which bots actually visit
Update your robots.txt as new AI bots emerge
Block only specific sensitive directories, not entire site
Robots.txt Complete Guide
Full guide to robots.txt configuration
AI Bot Access
Understanding bot access for GEO
Technical SEO Checklist
Complete technical optimization checklist