GEO-Score
ServicesAboutLearnLeaderboardReportsPricingBlog

GEO-Score

The Best GEO Analysis Tool

Test your content for AI search engines like ChatGPT, Perplexity, and Claude. Get actionable insights to improve your visibility.

Get GEO content tips

Resources

  • Knowledge Base
  • What is GEO?
  • GEO Blog
  • GEO Score Checker
  • Leaderboard
  • AI Glossary
  • Case Studies
  • GEO Insights
  • Reports

GEO Tools

  • Complete Website GEO Audit
  • GEO Analysis
  • AI Visibility Checker
  • ROI Calculator
  • Schema Validator
  • Perplexity SEO Checker
  • ChatGPT SEO Checker
  • Google AI Mode Checker
  • AI Crawler Checker
  • Agent Readiness Checker
  • llms.txt Generator
  • Query Fan-Out Tool
  • AI Bot Log Analyzer
  • Competitor Comparison
  • Professional Services
  • All Tools

Analyze Your

  • Blog Posts
  • Product Pages
  • Landing Pages
  • Homepages
  • Service Pages

GEO-Score

  • About
  • Privacy Policy
  • Terms & Conditions
  • Contact
  • White Label
  • Build Your Success

Follow GEO-Score

AI Search Guides

AI Search OptimizationPerplexity SEOChatGPT SEOLLM SEOAI Visibility CheckerAll guides →
GDPR CompliantData Hosted in EUMade in the Netherlands
KVK: 61855111BTW: NL004509498B76© 2026 GEO-score.onlineGEO-Score is part of Be-Found.
Empower your blogging with Bloffee
  1. Home
  2. /Glossary
  3. /Googlebot

AI crawlers

Googlebot

Googlebot is the generic name for Google Search’s two web crawlers, Googlebot Smartphone and Googlebot Desktop.

In plain words

Both follow the same robots.txt token, Googlebot, so you can’t target one of them on its own.

For most sites, Google mainly indexes the mobile version, so most crawl requests come from the smartphone crawler.

Example

The rule below keeps Googlebot out of a checkout folder, while the rest of the site stays crawlable. Remember that a robots.txt block stops crawling, not indexing: use noindex to keep a page out of results.

Keep Googlebot out of one folder:
User-agent: Googlebot
Disallow: /checkout/

Why it matters for AI search

Google says AI is built into Search, which is why robots.txt rules for Googlebot are the control for how your site is crawled for Search, AI Overviews and AI Mode included.

Blocking Googlebot affects Google Search, including Discover and all Search features, as well as Google Images, Google Video and Google News.

What you can do

  • Don’t block Googlebot if you want to appear in Google’s AI features; there is no separate Google AI crawler to allow.
  • Use nosnippet or max-snippet to limit what Google shows, AI Overviews included.
  • Verify real Googlebot traffic before you block anything: Google warns its user agent is often spoofed.

Try it: Free robots.txt generator

Often confused with

Google-Extended
Google-Extended is not a crawler and doesn’t affect Search. It only controls Gemini training and grounding.

Quick facts

Run by
Google
robots.txt name
Googlebot
Official IP list
static/crawling/ipranges/common-crawlers.json

Sources

  • Google Search Central: Googlebot
  • Google Search Central: AI features and your website
  • Google: Google's common crawlers

Facts and sources last checked on 19 October 2026.

Related terms

  • Google-Extended
  • nosnippet
  • noindex
  • Crawl budget
  • Bingbot

Go deeper on GEO-Score

  • robots.txt guide
  • Guide: Google AI Overviews

How does your page do in AI search?

GEO-Score checks one page on 22 signals, from AI crawler access to structured data, and shows what to fix first.

Analyze a page