At a glance
| robots.txt name | Google-Extended |
|---|---|
| Company | |
| Kind | Model training. Use in Gemini models. A rule only; it never visits |
| Visits sites | No. It is a robots.txt rule only |
| Official page | developers.google.com |
Google-Extended is a user agent token that site owners can use in robots.txt to control whether their content is used for training Gemini models. It does not have its own crawler; instead, it works with Googlebot, which does the actual crawling. When you set a rule for Google-Extended, it applies to the use of your content for AI training, not to Google Search. This gives you a way to manage how your content contributes to AI models while still allowing Google to index your site for search.
For businesses, deciding whether to block Google-Extended depends on your goals. If you want to limit how your content is used in AI training, blocking it is a reasonable choice. However, blocking Google-Extended does not affect your site's presence in Google Search or AI search features like AI Overviews, because those are governed by Googlebot. You can block Google-Extended without losing search visibility. To block it, add a group to robots.txt with 'User-agent: Google-Extended' and 'Disallow: /'. To allow it, you can use 'Allow: /' or simply omit any rule for it. Remember that robots.txt is a request, not a lock, and reputable crawlers follow it.
robots.txt lines
User-agent: Google-Extended Disallow: /
User-agent: Google-Extended Allow: /
Questions
What is Google-Extended?
Google-Extended is a control from Google that lets site owners opt out of having their content used to train Gemini models. It is not a crawler that visits sites; it is a rule that works alongside Googlebot.
Should I block Google-Extended?
Blocking Google-Extended is a common, reasonable choice if you do not want your content used for AI training. It does not stop AI search crawlers or affect your Google Search presence.
How do I block Google-Extended in robots.txt?
To block Google-Extended for your whole site, add these two lines to robots.txt: User-agent: Google-Extended Disallow: /
Does blocking Google-Extended affect AI search or Google?
No. Blocking Google-Extended does not affect AI search or Google Search, because those are governed by Googlebot. Google-Extended only controls the use of your content for training Gemini models.