Crawl and index controls
Web Bot Auth
Web Bot Auth is an authentication method in which a bot signs its HTTP requests cryptographically, so a website can verify that a request really comes from that bot.
In plain words
Anyone can write "GPTBot" in a user agent. A signature is much harder to fake.
It relies on two IETF drafts: one for a directory where a crawler shares its public keys, and one that defines how those keys attach the crawler’s identity to HTTP requests.
Example
Cloudflare uses Web Bot Auth as one way to verify bots and agents. A signed request can be checked against the crawler’s published keys before it is let through.
Why it matters for AI search
Fake crawlers are common: Google and Common Crawl both warn that their crawler names get spoofed.
Signed requests make it easier to welcome the real AI crawlers and keep impostors out.
What you can do
- If you use Cloudflare, look at its bot verification settings.
- Until signatures are everywhere, verify bots with the IP lists their operators publish.
- Don’t judge a crawler by its name alone; check that it is who it says it is.
Often confused with
- User agent
- A user agent is just a name the visitor sends. Web Bot Auth adds proof.
- IP verification
- An IP list shows where a request comes from. A signature shows who sent it.
Sources
Facts and sources last checked on 19 October 2026.
Related terms
Go deeper on GEO-Score
How does your page do in AI search?
GEO-Score checks one page on 22 signals, from AI crawler access to structured data, and shows what to fix first.
Analyze a page