Our own research
GEO experiments
We test how websites treat AI crawlers, on fresh samples, with the method written down first. Here is what we found, and what we did not.
Measured on October 20, 2026
The experiments
Results published
How many websites block AI crawlers?
We read the robots.txt of 6,596 sites and worked out, per AI bot, who may visit the homepage. Plus: who serves a real llms.txt file.
See the resultsResults published
The JavaScript blind spot
Some AI tools read a page without running its JavaScript. We compared what they get with what a browser shows, on 1,804 homepages.
See the resultsPlanned, not run yet
What can AI assistants actually read on a page?
A test page with secret code words in different places: plain text, JavaScript, structured data. We then ask ChatGPT, Gemini, Claude and others which words they can find. This needs its own test site first, so it has not run yet.
How we run every experiment
Studies like these are easy to get wrong. These are the rules we hold ourselves to.
- We write the plan down before we collect any data.
- We draw a fresh random sample from a public list, with a fixed seed, so others can draw the same sites.
- We show every site we could not measure, and why. Failures stay in the tables.
- Every percentage comes with its 95% range, per group of sites. We never mix the groups into one number.
- Every number has a date. The web changes, so we re-run instead of updating numbers silently.
- We publish totals and a CSV, never a list of named sites.
- We report what we measured. We don't claim that something makes you rank or get cited.
- Our crawler names itself, follows robots.txt and takes it slow.
More research
Our other research is based on GEO scores from our own audits. All research reports (English)
Check your own site
These are averages. Your own site is either in or out. Our free tools check it in seconds.
Sources of the site lists
Popular sites: the Tranco list 467YX (Le Pochat et al., NDSS 2019).
Dutch- and German-language sites: with content from Curlie.org - the largest human-edited directory of the web. Contribute by submitting a website or becoming an editor. Curlie data is licensed under CC BY 3.0.