At a glance
| robots.txt name | Meta-ExternalAgent |
|---|---|
| Company | Meta |
| Kind | Model training. Collects pages to train Meta's AI models |
| Visits sites | Yes |
| Official page | developers.facebook.com |
Meta-ExternalAgent collects pages from the web to train Meta's AI models. It is a training crawler, meaning its purpose is to gather data for model training rather than to index pages for search results. For businesses, this matters because the content it collects may be used to improve AI systems that could compete with or complement your own services. Blocking it is a common, reasonable choice and does not stop AI search crawlers.
Whether to block or allow Meta-ExternalAgent depends on your goals. If you want to limit how your content is used for AI training, blocking it is a sensible step. If you are comfortable with your content being used for training, you can allow it. Remember that robots.txt is a request, not a lock, and reputable crawlers follow it. A firewall or bot-protection service can block a crawler even when robots.txt allows it.
robots.txt lines
User-agent: Meta-ExternalAgent Disallow: /
User-agent: Meta-ExternalAgent Allow: /
Questions
What is Meta-ExternalAgent?
Meta-ExternalAgent is a crawler from Meta that collects pages to train Meta's AI models.
Should I block Meta-ExternalAgent?
Blocking it is a common, reasonable choice and does not stop AI search crawlers. It depends on whether you want your content used for AI training.
How do I block Meta-ExternalAgent in robots.txt?
To block it for the whole site, add these two lines to robots.txt: User-agent: Meta-ExternalAgent Disallow: /
Does blocking Meta-ExternalAgent affect AI search or Google?
No, blocking Meta-ExternalAgent does not stop AI search crawlers. It is a training crawler, so blocking it does not affect Google search.