What happens if I block all AI bots?

Blocking all AI bots means your content won't appear in ChatGPT, Claude, Perplexity, or Gemini search results. You could lose up to 40% of potential traffic from AI-powered search. We recommend allowing at least the critical AI search bots.

Will this affect my Google ranking?

Blocking AI bots (like GPTBot or ClaudeBot) does not directly affect your Google Search ranking. Google Search uses Googlebot, which is separate. However, blocking Google-Extended will prevent your content from appearing in Google Gemini AI responses.

How do I test my robots.txt file?

After downloading, upload it to the root of your website (yoursite.com/robots.txt). You can verify it works using Google Search Console's robots.txt tester, or by visiting the URL directly in your browser.

Can I block AI training but allow AI search?

Yes! Use the 'Privacy First' template. It blocks training-focused bots like CCBot and Google-Extended while allowing search bots like PerplexityBot and ClaudeBot to access your content for search results.

robots.txt is a plain text file placed in your website's root directory that tells web crawlers and bots which pages they can or cannot access. It's the standard way to control bot access across the web.

How often should I update my robots.txt?

Review your robots.txt whenever you add new sections to your site, launch new features, or when new AI bots appear. We recommend checking it quarterly at minimum, as the AI bot landscape evolves quickly.

What happens if I block all AI bots?

Blocking all AI bots means your content won't appear in ChatGPT, Claude, Perplexity, or Gemini search results. You could lose up to 40% of potential traffic from AI-powered search. We recommend allowing at least the critical AI search bots.

Will this affect my Google ranking?

Blocking AI bots (like GPTBot or ClaudeBot) does not directly affect your Google Search ranking. Google Search uses Googlebot, which is separate. However, blocking Google-Extended will prevent your content from appearing in Google Gemini AI responses.

How do I test my robots.txt file?

After downloading, upload it to the root of your website (yoursite.com/robots.txt). You can verify it works using Google Search Console's robots.txt tester, or by visiting the URL directly in your browser.

Can I block AI training but allow AI search?

Yes! Use the 'Privacy First' template. It blocks training-focused bots like CCBot and Google-Extended while allowing search bots like PerplexityBot and ClaudeBot to access your content for search results.

robots.txt is a plain text file placed in your website's root directory that tells web crawlers and bots which pages they can or cannot access. It's the standard way to control bot access across the web.

How often should I update my robots.txt?

Review your robots.txt whenever you add new sections to your site, launch new features, or when new AI bots appear. We recommend checking it quarterly at minimum, as the AI bot landscape evolves quickly.

AI Bot Control

robots.txt Generator for AI Bots

Name: robots.txt Generator for AI Bots
Author: GEO-Score

Generate optimized robots.txt files with support for 40+ AI bots including ChatGPT, Claude, Perplexity, and Gemini. Control which bots can access your content.

40+

Bots supported

Languages

Free

No credit card required

Choose a Starting Template

Configure Bot Access

critical

GPTBotOpenAI

OpenAI ChatGPT web crawler

critical

ChatGPT-UserOpenAI

ChatGPT Browse with Bing

critical

Google-ExtendedGoogle

Google Gemini AI crawler

critical

PerplexityBotPerplexity

Perplexity AI search engine

high

ClaudeBotAnthropic

Anthropic Claude web crawler

high

Claude-WebAnthropic

Claude web search feature

medium

BLEXBotBLEXBot

Aggressive content scraper

medium

MegaIndexMegaIndex

Spam indexer bot

medium

SeznamBotSeznam

Aggressive crawler

medium

linkdexbotLinkdex

Deprecated link crawler

medium

PetalBotHuawei

Aggressive Huawei crawler

Protected Paths

disallow/admin/

disallow/api/

disallow/dashboard/

disallow/auth/

disallow/login/

Sitemap URL

Usually at https://yoursite.com/sitemap.xml — check by visiting that URL. This tells bots where to find all your pages.

robots.txt23 lines

1# ==========================================================================
2# robots.txt
3# Generated by GEO-Score robots.txt Generator
4# https://geo-score.online/tools/robots-generator
5# Generated: 2026-08-26
6# ==========================================================================
7 
8# Blocked bots
9User-agent: BLEXBot
10User-agent: MegaIndex
11User-agent: SeznamBot
12User-agent: linkdexbot
13User-agent: PetalBot
14Disallow: /
15 
16# Default rules for all other bots
17User-agent: *
18Disallow: /admin/
19Disallow: /api/
20Disallow: /dashboard/
21Disallow: /auth/
22Disallow: /login/
23

No sitemap URL declared

Add your sitemap URL (e.g. https://yoursite.com/sitemap.xml) above. This is the bridge between robots.txt and your sitemap — it helps all bots discover your pages.

Frequently Asked Questions

Choose a Starting Template

Configure Bot Access

critical

GPTBotOpenAI

OpenAI ChatGPT web crawler

critical

ChatGPT-UserOpenAI

ChatGPT Browse with Bing

critical

Google-ExtendedGoogle

Google Gemini AI crawler

critical

PerplexityBotPerplexity

Perplexity AI search engine

high

ClaudeBotAnthropic

Anthropic Claude web crawler

high

Claude-WebAnthropic

Claude web search feature

medium

BLEXBotBLEXBot

Aggressive content scraper

medium

MegaIndexMegaIndex

Spam indexer bot

medium

SeznamBotSeznam

Aggressive crawler

medium

linkdexbotLinkdex

Deprecated link crawler

medium

PetalBotHuawei

Aggressive Huawei crawler

1# ==========================================================================

2# robots.txt

3# Generated by GEO-Score robots.txt Generator

4# https://geo-score.online/tools/robots-generator

5# Generated: 2026-08-26

6# ==========================================================================

8# Blocked bots

9User-agent: BLEXBot

10User-agent: MegaIndex

11User-agent: SeznamBot

12User-agent: linkdexbot

13User-agent: PetalBot

14Disallow: /

16# Default rules for all other bots

17User-agent: *

18Disallow: /admin/

19Disallow: /api/

20Disallow: /dashboard/

21Disallow: /auth/

22Disallow: /login/

robots.txt Generator for AI Bots

Choose a Starting Template

Maximum Visibility

Privacy First

E-commerce Optimized

Block All AI

Configure Bot Access

AI Search Engines

AI Training Bots

Traditional Search Engines

Social Media Bots

SEO Tools

Bad Bots

Protected Paths

Frequently Asked Questions

robots.txt Generator for AI Bots

Choose a Starting Template

Maximum Visibility

Privacy First

E-commerce Optimized

Block All AI

Configure Bot Access

AI Search Engines

AI Training Bots

Traditional Search Engines

Social Media Bots

SEO Tools

Bad Bots

Protected Paths

Frequently Asked Questions