At a glance
| robots.txt name | Claude-SearchBot |
|---|---|
| Company | Anthropic |
| Kind | AI search. Finds pages for Claude's web search |
| Visits sites | Yes |
| Official page | support.claude.com |
Claude-SearchBot visits sites to find pages for Claude's web search. It reads content so the assistant can include it in answers, with quotes, citations and links. If you block it, Claude usually cannot cite or link to your site in those answers. That can reduce your visibility when people ask Claude questions your business could answer.
You can allow or block Claude-SearchBot in robots.txt. Blocking is a request, not a lock; reputable crawlers follow it, but a firewall or bot-protection service can enforce a block. To block it for the whole site, add a group with its User-agent line and Disallow: /. To allow it, use Allow: / or no rule for it. Being readable does not mean being recommended; Vasukai's free checker reads your robots.txt and shows whether this crawler may read a page, and tests how the server answers.
robots.txt lines
User-agent: Claude-SearchBot Disallow: /
User-agent: Claude-SearchBot Allow: /
Questions
What is Claude-SearchBot?
Claude-SearchBot is a search crawler from Anthropic. It finds and reads pages so Claude can quote, cite and link to them in answers that search the web.
Should I block Claude-SearchBot?
It depends on whether you want Claude to cite and link to your site. Blocking it usually means the assistant cannot cite or link to the site, which can reduce your visibility in AI search. If you want that visibility, allow it.
How do I block Claude-SearchBot in robots.txt?
Add these two lines to robots.txt: User-agent: Claude-SearchBot and Disallow: /. This is a request, not a lock; a firewall or bot-protection service can block the crawler even when robots.txt allows it.
Does blocking Claude-SearchBot affect AI search or Google?
Blocking Claude-SearchBot affects Claude's web search, not Google. It usually means Claude cannot cite or link to your site in its answers. Google uses its own crawlers, so blocking Claude-SearchBot does not change how Google crawls or ranks your site.