At a glance
| robots.txt name | Claude-User |
|---|---|
| Company | Anthropic |
| Kind | Fetch on request. Opens a page when someone asks Claude about it |
| Visits sites | Yes |
| Official page | support.claude.com |
Claude-User opens a page when someone asks Claude about it. This means it does not crawl the web on its own; it only fetches pages that a user specifically requests. For a business, this matters because it controls whether Claude can access and potentially use your content in its responses. If you block Claude-User, Claude may not be able to read your page when a user asks about it, which could affect how your business is represented in those interactions.
Whether to block or allow Claude-User depends on your goals. If you want Claude to be able to answer questions about your pages, you should allow it. If you prefer that Claude does not access your content on request, you can block it. Remember that robots.txt is a request, not a lock; reputable crawlers follow it, but a firewall or bot-protection service can block a crawler even when robots.txt allows it.
robots.txt lines
User-agent: Claude-User Disallow: /
User-agent: Claude-User Allow: /
Questions
What is Claude-User?
Claude-User is a user fetcher from Anthropic. It opens a page only when a person asks the assistant about that page or a link. It is not a general crawl.
Should I block Claude-User?
It depends on whether you want Claude to access your pages when users ask about them. If you want Claude to be able to answer questions about your content, allow it. If you prefer that it does not, block it. Blocking may affect how your business is represented in those interactions.
How do I block Claude-User in robots.txt?
To block Claude-User for the whole site, add these two lines to robots.txt: User-agent: Claude-User Disallow: /
Does blocking Claude-User affect AI search or Google?
Blocking Claude-User only affects whether Claude can fetch your pages on request. It does not affect Google search. Being readable by AI does not mean being recommended; AI assistants may still recommend other businesses. Vasukai's free checker can show whether this crawler may read a page and test how the server answers.