AI crawlers
Meta-ExternalAgent
Also called: Meta-ExternalFetcher
Meta-ExternalAgent is Meta’s web crawler for uses such as training foundation AI models or improving products by indexing content directly.
In plain words
Its user agent looks like meta-externalagent/1.1.
Meta also runs Meta-ExternalFetcher. It fetches single links at a user’s request, for example to help AI complete tasks, and Meta says it may bypass robots.txt.
Example
A site that doesn’t want Meta to crawl its content for these uses adds the group below to robots.txt.
User-agent: meta-externalagent
Disallow: /Why it matters for AI search
One crawler covers several uses, so blocking it affects both AI training and Meta’s product indexing.
Meta-ExternalFetcher works for individual users and may not follow robots.txt.
What you can do
- Decide about Meta-ExternalAgent separately from Meta’s other crawlers, such as facebookexternalhit.
- Use robots.txt for Meta-ExternalAgent.
- Keep private content behind a login, because Meta-ExternalFetcher may bypass robots.txt.
Try it: Free robots.txt generator
Often confused with
- Meta-ExternalFetcher
- Meta-ExternalFetcher fetches one link at a user’s request. Meta-ExternalAgent crawls the web.
- facebookexternalhit
- facebookexternalhit is another Meta crawler. Meta says it might bypass robots.txt when it performs security or integrity checks.
Quick facts
- Run by
- Meta
- robots.txt name
- meta-externalagent, meta-externalfetcher
Sources
Facts and sources last checked on 19 October 2026.
Related terms
How does your page do in AI search?
GEO-Score checks one page on 22 signals, from AI crawler access to structured data, and shows what to fix first.
Analyze a page