At a glance
| robots.txt name | GPTBot |
|---|---|
| Company | OpenAI |
| Kind | Model training. Collects pages to train OpenAI's models |
| Visits sites | Yes |
| User agent | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.1; +https://openai.com/gptbot |
| Official page | platform.openai.com |
GPTBot collects pages that may be used to train OpenAI's models. It visits sites and follows robots.txt. Blocking it is a common, reasonable choice and does not stop AI search crawlers.
For a business, allowing GPTBot means your content may help train future AI models, but it does not mean AI assistants will recommend you. Blocking it keeps your content out of training but does not affect AI search or Google. You can allow or block it in robots.txt.
robots.txt lines
User-agent: GPTBot Disallow: /
User-agent: GPTBot Allow: /
Questions
What is GPTBot?
GPTBot is OpenAI's crawler that collects pages to train its models.
Should I block GPTBot?
Blocking GPTBot is a common, reasonable choice. It does not stop AI search crawlers. Whether to block depends on if you want your content used for training.
How do I block GPTBot in robots.txt?
To block GPTBot for the whole site, add these two lines to robots.txt: User-agent: GPTBot Disallow: /
Does blocking GPTBot affect AI search or Google?
No. Blocking GPTBot does not affect AI search or Google. It only stops this training crawler.