Free tools · AI crawlers
AI crawlers: the full list
18 crawlers from 12 companies, grouped by what they do. Open one for its user agent, the robots.txt lines to block or allow it, and a live check on your site.
AI search
Find pages for AI answers that search the web. Block these and the assistant can't cite or link to you.
| Crawler | Company | What it does | Visits sites |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Finds pages for ChatGPT search answers | Yes |
| Claude-SearchBot | Anthropic | Finds pages for Claude's web search | Yes |
| PerplexityBot | Perplexity | Finds pages for Perplexity answers | Yes |
| Googlebot | Google Search, which AI Overviews and AI Mode draw on | Yes | |
| Bingbot | Microsoft | Bing search, which Copilot draws on | Yes |
| DuckAssistBot | DuckDuckGo | Finds pages for DuckDuckGo's AI answers | Yes |
Fetch on request
Open a page only when someone asks the assistant about it.
| Crawler | Company | What it does | Visits sites |
|---|---|---|---|
| ChatGPT-User | OpenAI | Opens a page when someone asks ChatGPT about it | Yes |
| Claude-User | Anthropic | Opens a page when someone asks Claude about it | Yes |
| Perplexity-User | Perplexity | Opens a page when someone asks Perplexity about it | Yes |
| MistralAI-User | Mistral | Opens a page when someone asks Le Chat about it | Yes |
Model training
Collect pages that may be used to train AI models. Blocking them doesn't stop AI search.
| Crawler | Company | What it does | Visits sites |
|---|---|---|---|
| GPTBot | OpenAI | Collects pages to train OpenAI's models | Yes |
| ClaudeBot | Anthropic | Collects pages to train Claude | Yes |
| Google-Extended | Use in Gemini models. A rule only; it never visits | No, a rule only | |
| Applebot-Extended | Apple | Use in Apple's AI models. A rule only; it never visits | No, a rule only |
| Meta-ExternalAgent | Meta | Collects pages to train Meta's AI models | Yes |
| Amazonbot | Amazon | Collects pages for Amazon's AI, including Alexa | Yes |
| Bytespider | ByteDance | Collects pages to train ByteDance's models | Yes |
| CCBot | Common Crawl | The open web archive many AI models are trained on | Yes |
Readable isn’t the same as recommended
The free AI check shows whether ChatGPT names your business and who it names instead. No card needed.