Our own research
How many websites block AI crawlers?
On October 20, 2026, 11.7% of the 10,000 most popular websites had a robots.txt rule that keeps OpenAI's training bot GPTBot off their homepage. Fewer had one for the bot behind ChatGPT search: 6.3% had a rule against OAI-SearchBot. Another 1.1% answered with an error instead of a robots.txt file. That is more likely a server problem than a choice, but it keeps bots out too.
Measured on October 20, 2026
- of the top 10,000 sites have a robots.txt rule against GPTBot
- 11.7%
- have a rule against OAI-SearchBot (ChatGPT search)
- 6.3%
- serve a text file at /llms.txt
- 13.4%
What we tested
Almost every website has a small file called robots.txt. It tells bots which pages they may visit. AI companies say their bots read it, so it is the main way a site says yes or no to them.
We fetched that file from every site in our sample and read it the way the official standard (RFC 9309) says bots must. For each AI bot we asked one question: may it open the homepage?
A file can say no in two ways: with a rule that names the bot, or with a rule for all bots (User-agent: *). And when the file itself gives a server error, the standard says bots must stay away completely. We count all three, and show them apart.
Which AI bots are blocked, and where
Each table shows, per bot, the share of sites whose robots.txt keeps it off the homepage. Popular sites block AI bots about three times as often as Dutch- and German-language directory sites. On popular sites, training bots are blocked more often than the bots that fetch pages for AI search.
| Bot | Top 10,000 | 10,001–100,000 | 100,001–1 million | Dutch-language | German-language |
|---|---|---|---|---|---|
| GPTBot | 12.9%11.1%–14.9% | 10.8%9.2%–12.6% | 12.6%10.9%–14.6% | 3.6%2.8%–4.7% | 3.8%3%–5% |
| ClaudeBot | 12.6%10.9%–14.6% | 9.1%7.6%–10.8% | 12.2%10.6%–14.2% | 3.7%2.8%–4.8% | 3.1%2.3%–4.1% |
| CCBot | 14%12.2%–16.1% | 9.6%8.1%–11.4% | 9.6%8.1%–11.3% | 3.3%2.5%–4.3% | 3.3%2.5%–4.3% |
| Google-Extended | 11.1%9.4%–13% | 7.7%6.4%–9.3% | 11.1%9.5%–13% | 2.9%2.2%–3.9% | 3%2.2%–4% |
| Applebot-Extended | 11.2%9.6%–13.1% | 8.1%6.7%–9.7% | 11.2%9.6%–13.1% | 2.6%1.9%–3.6% | 2.9%2.2%–3.9% |
| meta-externalagent | 10.9%9.3%–12.8% | 8.4%7%–10.1% | 11.2%9.6%–13.1% | 3.1%2.3%–4.2% | 3.6%2.8%–4.7% |
| Bytespider | 13.7%11.9%–15.7% | 9.8%8.3%–11.5% | 9.1%7.6%–10.8% | 4.1%3.1%–5.2% | 3.6%2.8%–4.7% |
| Amazonbot | 10.8%9.1%–12.6% | 8.1%6.7%–9.7% | 12.1%10.4%–14% | 3.8%2.9%–4.9% | 3%2.2%–4% |
| Sites counted (n) | 1,227 | 1,270 | 1,266 | 1,403 | 1,430 |
| Bot | Top 10,000 | 10,001–100,000 | 100,001–1 million | Dutch-language | German-language |
|---|---|---|---|---|---|
| OAI-SearchBot | 7.4%6.1%–9% | 4.7%3.7%–6% | 8.4%7%–10% | 2.6%1.9%–3.6% | 2.4%1.8%–3.4% |
| ChatGPT-User | 8.5%7%–10.2% | 6.6%5.4%–8.1% | 9.6%8.1%–11.4% | 3%2.2%–4% | 2.8%2.1%–3.8% |
| Claude-SearchBot | 6.8%5.5%–8.3% | 4.5%3.5%–5.8% | 8.3%6.9%–9.9% | 2.6%1.9%–3.5% | 2.4%1.8%–3.4% |
| Claude-User | 6.7%5.4%–8.2% | 4.6%3.6%–5.9% | 8.3%6.9%–9.9% | 2.4%1.7%–3.4% | 2.4%1.7%–3.3% |
| PerplexityBot | 9.9%8.3%–11.7% | 5.5%4.4%–6.9% | 9.3%7.8%–11% | 2.8%2%–3.8% | 2.7%1.9%–3.6% |
| Perplexity-User | 7.3%6%–8.9% | 4.3%3.3%–5.5% | 8.1%6.8%–9.8% | 2.6%1.9%–3.5% | 2.6%1.9%–3.5% |
| Sites counted (n) | 1,227 | 1,270 | 1,266 | 1,403 | 1,430 |
| Bot | Top 10,000 | 10,001–100,000 | 100,001–1 million | Dutch-language | German-language |
|---|---|---|---|---|---|
| Googlebot | 3.6%2.7%–4.8% | 2.4%1.7%–3.4% | 3.6%2.7%–4.8% | 1.8%1.2%–2.6% | 2.2%1.5%–3.1% |
| bingbot | 3.9%3%–5.1% | 3.3%2.5%–4.4% | 4.5%3.5%–5.8% | 2.4%1.7%–3.3% | 2.4%1.8%–3.4% |
| Sites counted (n) | 1,227 | 1,270 | 1,266 | 1,403 | 1,430 |
Under each share you see its range: with 95% confidence the real share lies between those two numbers. Columns 1-3 are popular sites by rank in the Tranco list; columns 4-5 are Dutch- and German-language sites from the Curlie directory.
How sites say no
Of the top 10,000 sites that keep GPTBot out, 69% use a rule that names GPTBot. For 22.2%, only the rule for all bots does it, maybe without the owner ever thinking of AI. For 8.9%, robots.txt gives a server error, which counts as a full block.
This matters when you count. A check that only looks for a bot's own name misses the second group. An earlier page of ours relied on exactly such a check, so we took it down and measured again.
| Bot | Top 10,000 | 10,001–100,000 | 100,001–1 million | Dutch-language | German-language |
|---|---|---|---|---|---|
| A rule that names GPTBot | 8.9%7.4%–10.6% | 7.6%6.2%–9.1% | 5.4%4.3%–6.8% | 1.4%0.9%–2.2% | 1.6%1.1%–2.4% |
| Only the rule for all bots (*) | 2.9%2.1%–3.9% | 2.2%1.5%–3.2% | 5.7%4.5%–7.1% | 1.3%0.8%–2% | 1.3%0.9%–2.1% |
| robots.txt gives a server error | 1.1%0.7%–1.9% | 1%0.6%–1.7% | 1.6%1%–2.4% | 0.9%0.5%–1.6% | 0.9%0.5%–1.5% |
| Sites counted (n) | 1,227 | 1,270 | 1,266 | 1,403 | 1,430 |
Training bots versus search bots
OpenAI uses two bots. GPTBot collects content for training, OAI-SearchBot finds pages for ChatGPT search. Of the top 10,000 sites that block GPTBot, 42.4% still let OAI-SearchBot in.
That share includes sites whose robots.txt gave an error, which keeps OAI-SearchBot out as well. Among the sites with a rule against GPTBot, 46.5% let OAI-SearchBot in (exploratory).
That looks like a deliberate choice. OpenAI says sites that block OAI-SearchBot are not shown in ChatGPT search answers, while blocking GPTBot alone does not do that.
| Bot | Top 10,000 | 10,001–100,000 | 100,001–1 million | Dutch-language | German-language |
|---|---|---|---|---|---|
| Block at least one AI bot | 17.2%15.2%–19.4% | 13.3%11.5%–15.3% | 15.5%13.6%–17.6% | 5.2%4.2%–6.5% | 4.4%3.5%–5.6% |
| Block every AI bot on our list | 4.3%3.3%–5.6% | 3.1%2.3%–4.2% | 4%3.1%–5.3% | 2.3%1.6%–3.2% | 2.2%1.6%–3.1% |
| Sites counted (n) | 1,227 | 1,270 | 1,266 | 1,403 | 1,430 |
| Bot | Top 10,000 | 10,001–100,000 | 100,001–1 million | Dutch-language | German-language |
|---|---|---|---|---|---|
| Still allow OAI-SearchBot | 42.4%35%–50.2% | 56.2%47.8%–64.2% | 33.8%26.9%–41.4% | 29.4%18.7%–43% | 36.4%24.9%–49.6% |
| Sites that block GPTBot (n) | 158 | 137 | 160 | 51 | 55 |
llms.txt: who has one?
An llms.txt file is a short summary of a website for AI tools, written in Markdown. It is a proposal, not a standard. Google says Google Search ignores it. Read more in our llms.txt guide, or check your own file with our llms.txt checker.
13.4% of the top 10,000 sites serve a text file at /llms.txt. Watch out for look-alikes: 13.2% answer with a normal web page instead. Of the text files, 83.8% pass Chrome Lighthouse's basic llms.txt check.
| llms.txt | Top 10,000 | 10,001–100,000 | 100,001–1 million | Dutch-language | German-language |
|---|---|---|---|---|---|
| Serve a real llms.txt text file | 13.4%11.6%–15.5% | 12.4%10.6%–14.4% | 13.5%11.7%–15.6% | 5.8%4.7%–7.2% | 7.4%6.1%–8.9% |
| Serve a text file that passes Lighthouse's check | 11.2%9.5%–13.2% | 10.6%9%–12.5% | 11.4%9.7%–13.4% | 5.4%4.4%–6.8% | 7%5.8%–8.5% |
| Serve a web page instead (not a real file) | 13.2%11.3%–15.2% | 13.5%11.6%–15.5% | 13.9%12.1%–16.1% | 12.7%11.1%–14.6% | 9.6%8.2%–11.3% |
| Sites that answered (n) | 1,148 | 1,196 | 1,147 | 1,360 | 1,375 |
| llms.txt | Top 10,000 | 10,001–100,000 | 100,001–1 million | Dutch-language | German-language |
|---|---|---|---|---|---|
| Pass Chrome Lighthouse's llms.txt check | 83.8%77.1%–88.8% | 85.8%79.3%–90.5% | 84.5%78%–89.4% | 93.7%86%–97.3% | 94.1%87.8%–97.3% |
| Follow the llmstxt.org format without errors | 49.4%41.6%–57.2% | 48.6%40.7%–56.6% | 27.7%21.3%–35.3% | 54.4%43.5%–65% | 55.9%46.2%–65.1% |
| Sites with a real file (n) | 154 | 148 | 155 | 79 | 102 |
Most files that break the llmstxt.org format list pages without a link, or don't start with a title line. Some "files" are only an error message or the word ok.
Other studies report other numbers because they count differently. The Web Almanac 2025 found a valid llms.txt on about 2.1% of all the sites it crawls. Ahrefs (vendor data) counted 28% of 137,210 domains from its own analytics customers, an upper bound by its own account, and found that 97% of those files got no requests in May 2026. SE Ranking (vendor data) found an llms.txt on 10% of about 300,000 domains.
What this does not prove
- robots.txt is a request, not a lock. Many sites also stop bots with a firewall or a CDN setting. We only read what the file says.
- We are not GPTBot. A site can answer AI bots differently than it answered our bot, and some sites block unknown bots completely.
- We only checked access to the homepage. A site can allow the homepage and block everything else.
- This is one snapshot from one day. It does not show a trend. Sites change their files all the time.
- It says nothing about whether blocking (or an llms.txt) changes how often AI tools show or quote a site.
How we did it
We drew 1,500 sites at random from each of five groups: three rank bands of the Tranco list 467YX of popular websites (ranks 1-10,000, 10,001-100,000 and 100,001-1,000,000), plus the Dutch- and German-language sections of the Curlie web directory (47,502 and 309,045 sites listed with their homepage). A fixed random seed means anyone with the same two files draws the same sites.
One computer in the Netherlands, on a normal home internet connection, visited every site in about 25 minutes. It announced itself honestly as GEOScoreBot/1.0 (+https://geo-score.online), paused between requests to the same site and only fetched what robots.txt allowed it to.
We read each robots.txt with our RFC 9309 parser, the same one behind our AI crawler checker. A group for a bot's own name wins over the rule for all bots. No robots.txt (a 4xx answer) means every bot may visit. A server error (5xx) means no bot may visit. RFC 9309 counts 429 (too many requests) as a 4xx, so bots may visit, but Google and Lighthouse read it as a no, and so do we. Sites that did not answer at all (no DNS or no connection) are left out of the shares. The standard says bots must then stay away completely too, but such an error tells us nothing about what the site wants.
The numbers at the top of this page count only robots.txt rules: a rule that names the bot, or the rule for all bots. We added that split after we registered our method, so treat it as exploratory. The tables keep the registered measure, which also counts an error answer as a block.
We wrote down the questions, the sample and every measure before the crawl started, and saved that plan. Anything we decided after seeing the data is marked as exploratory.
| Step | Top 10,000 | 10,001–100,000 | 100,001–1 million | Dutch-language | German-language |
|---|---|---|---|---|---|
| Sites drawn | 1,500 | 1,500 | 1,500 | 1,500 | 1,500 |
| No answer at all (no website, DNS or connection error) | 273 | 230 | 234 | 97 | 70 |
| robots.txt answered: counted in the shares | 1,227 | 1,270 | 1,266 | 1,403 | 1,430 |
| … a real robots.txt file | 897 | 913 | 898 | 1,012 | 1,014 |
| … a web page instead of a robots.txt file | 91 | 84 | 87 | 115 | 80 |
| … no robots.txt (4xx) | 225 | 260 | 261 | 263 | 323 |
| … server error (429 or 5xx) | 14 | 13 | 20 | 13 | 13 |
| Homepage opened normally for our bot | 922 | 1,017 | 983 | 1,311 | 1,350 |
| Homepage refused our bot (401, 403 or 429) | 175 | 147 | 113 | 35 | 16 |
| Homepage gave another answer or failed | 68 | 59 | 73 | 25 | 29 |
| Homepage not visited: robots.txt said no to us | 62 | 47 | 97 | 32 | 35 |
A homepage that refuses our bot is bot protection, not a robots.txt rule. We never count it as "blocks AI".
RFC 9309, the robots.txt standardOther studies on the same question
The Web Almanac 2025 (HTTP Archive) found GPTBot named in robots.txt on 4.5% of desktop sites, across millions of sites. In our sample of the top 10,000, 12.6% of the files name GPTBot. Naming a bot is not the same as blocking it: some files name a bot to allow it.
Cloudflare Radar checks the robots.txt files of the 10,000 most popular domains every week and counts rules that name each AI bot (vendor data). We also count the rule for all bots and server errors, and we look beyond the top 10,000.
A 2024 study by Longpre and others, Consent in Crisis, followed about 14,000 domains behind big AI training datasets. It found that restrictions on AI crawlers rose sharply in 2023 and 2024.
Check your own site
These are averages. Your own site is either in or out. Our free tools check it in seconds.
Download the data
Every number on this page is in the CSV file, one row per number, with its count, its base and its 95% range. It contains totals only, no site names.
Download the numbers (CSV)You may reuse the numbers under the CC BY 4.0 licence. Please name GEO-Score and link to this page.
Sources of the site lists
Popular sites: the Tranco list 467YX (Le Pochat et al., NDSS 2019).
Dutch- and German-language sites: with content from Curlie.org - the largest human-edited directory of the web. Contribute by submitting a website or becoming an editor. Curlie data is licensed under CC BY 3.0.
October 20, 2026: first measurement published.