What is a user-agent name?
Every bot sends a name with each request: its user-agent. In robots.txt you use the short form of that name, the token, to give a bot its own rules. Case doesn't matter: GPTBot and gptbot are the same name.
AI companies run different bots for different jobs. Searchers find pages for AI search, learners collect pages for training, and helpers open a page when someone asks about it. Google-Extended and Applebot-Extended are not crawlers at all: they only decide how pages Google and Apple already crawl may be used.
Blocking a learner keeps your pages out of future training. Blocking a searcher keeps them out of that product's answers. Want to see what your own robots.txt does? Try the free AI Crawler Checker.
AI bot names, checked against each company's documentation
| robots.txt name | Company | What it does |
|---|