Crawl and index controls
X-Robots-Tag
X-Robots-Tag is an HTTP response header that can carry any rule the robots meta tag can, such as noindex or nosnippet.
In plain words
A meta tag only works inside an HTML page. A header works for any file your server sends.
That makes it the way to control PDFs, images and video files.
Rules without a crawler name apply to all crawlers. Google says the header name and its values are not case sensitive.
Example
A company publishes its price lists as PDFs and doesn’t want them in Google. Its server sends the header below with each PDF.
HTTP/1.1 200 OK
Content-Type: application/pdf
X-Robots-Tag: noindexWhy it matters for AI search
Search engines can only follow it if they are allowed to crawl the file.
You can combine several rules in one header, separated by commas.
What you can do
- Use it for files without HTML, such as PDFs.
- Set it in your server or CDN configuration.
- After deploying, check the response headers in your browser’s developer tools.
Often confused with
- Robots meta tag
- Same rules, different place: the meta tag lives in the HTML, X-Robots-Tag in the HTTP response.
- robots.txt
- robots.txt controls crawling by path. X-Robots-Tag controls indexing and snippets per response.
Sources
- Google Search Central: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
- Google Search Central: Block Search indexing with noindex
Facts and sources last checked on 19 October 2026.
Related terms
Go deeper on GEO-Score
How does your page do in AI search?
GEO-Score checks one page on 22 signals, from AI crawler access to structured data, and shows what to fix first.
Analyze a page