At a glance
| robots.txt name | ClaudeBot |
|---|---|
| Company | Anthropic |
| Kind | Model training. Collects pages to train Claude |
| Visits sites | Yes |
| User agent | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com) |
| Official page | support.claude.com |
ClaudeBot visits sites to gather pages that may be used to train AI models. As a training crawler, it collects content for model development rather than for live search results. Blocking it does not stop AI search crawlers, so your site can still be read by assistants that answer questions.
For businesses, allowing ClaudeBot means your content may help train future AI models, but it does not mean that AI assistants will recommend you. Being readable by AI does not mean being recommended. If you prefer not to contribute to model training, blocking ClaudeBot is a straightforward decision. You can block it in robots.txt, though that is a request, not a lock. A firewall or bot-protection service can block it even if robots.txt allows it.
robots.txt lines
User-agent: ClaudeBot Disallow: /
User-agent: ClaudeBot Allow: /
Questions
What is ClaudeBot?
ClaudeBot is a crawler from Anthropic that collects pages to train Claude. It is a training crawler, not a search crawler.
Should I block ClaudeBot?
Blocking ClaudeBot is a common, reasonable choice if you do not want your content used for model training. It does not stop AI search crawlers, so your site can still be read by AI assistants that answer questions.
How do I block ClaudeBot in robots.txt?
Add these two lines to your robots.txt file: User-agent: ClaudeBot and Disallow: /. To allow it, use Allow: / or no rule for it.
Does blocking ClaudeBot affect AI search or Google?
No. ClaudeBot is a training crawler, so blocking it does not stop AI search crawlers. It also does not affect Google, which uses its own crawlers.