AI crawlers
Googlebot
Googlebot is the generic name for Google Search’s two web crawlers, Googlebot Smartphone and Googlebot Desktop.
In plain words
Both follow the same robots.txt token, Googlebot, so you can’t target one of them on its own.
For most sites, Google mainly indexes the mobile version, so most crawl requests come from the smartphone crawler.
Example
The rule below keeps Googlebot out of a checkout folder, while the rest of the site stays crawlable. Remember that a robots.txt block stops crawling, not indexing: use noindex to keep a page out of results.
User-agent: Googlebot
Disallow: /checkout/Why it matters for AI search
Google says AI is built into Search, which is why robots.txt rules for Googlebot are the control for how your site is crawled for Search, AI Overviews and AI Mode included.
Blocking Googlebot affects Google Search, including Discover and all Search features, as well as Google Images, Google Video and Google News.
What you can do
- Don’t block Googlebot if you want to appear in Google’s AI features; there is no separate Google AI crawler to allow.
- Use nosnippet or max-snippet to limit what Google shows, AI Overviews included.
- Verify real Googlebot traffic before you block anything: Google warns its user agent is often spoofed.
Try it: Free robots.txt generator
Often confused with
- Google-Extended
- Google-Extended is not a crawler and doesn’t affect Search. It only controls Gemini training and grounding.
Quick facts
- Run by
- robots.txt name
- Googlebot
- Official IP list
- static/crawling/ipranges/common-crawlers.json
Sources
- Google Search Central: Googlebot
- Google Search Central: AI features and your website
- Google: Google's common crawlers
Facts and sources last checked on 19 October 2026.
Related terms
How does your page do in AI search?
GEO-Score checks one page on 22 signals, from AI crawler access to structured data, and shows what to fix first.
Analyze a page