AI crawlers
CCBot
CCBot is the web crawler of Common Crawl, a non-profit foundation that builds an open repository of web crawl data anyone can access and analyze.
In plain words
Common Crawl says it was founded to democratize access to web information. It publishes its crawl data for free, so researchers, companies and AI developers can all download it.
CCBot identifies itself as CCBot/2.0 in its user agent.
Example
Common Crawl data has been used to train large language models. OpenAI’s 2020 GPT-3 paper, for example, says most of its training data came from Common Crawl, filtered for quality.
User-agent: CCBot
Disallow: /Why it matters for AI search
Because the data is open, one crawl can end up in many different datasets and models.
Common Crawl warns that some crawlers falsely identify themselves as CCBot.
What you can do
- Decide whether you want your pages in an open web archive that others reuse.
- If not, add the CCBot group shown above to your robots.txt.
- Verify suspicious CCBot traffic before trusting the name, as Common Crawl recommends.
Try it: Free robots.txt generator
Often confused with
- GPTBot
- GPTBot collects data for OpenAI only. CCBot collects data for a public archive that anyone can use.
Quick facts
- Run by
- Common Crawl
- robots.txt name
- CCBot
Sources
Facts and sources last checked on 19 October 2026.
Related terms
How does your page do in AI search?
GEO-Score checks one page on 22 signals, from AI crawler access to structured data, and shows what to fix first.
Analyze a page