At a glance
| robots.txt name | Googlebot |
|---|---|
| Company | |
| Kind | AI search. Google Search, which AI Overviews and AI Mode draw on |
| Visits sites | Yes |
| Official page | developers.google.com |
Googlebot is a search crawler that finds and reads pages so the assistant can quote, cite and link to them in answers that search the web. Blocking it usually means the assistant cannot cite or link to the site. It is owned by Google and is used for AI search.
Almost every business should allow Googlebot. Blocking it would remove the site from Google Search as well as from AI Overviews and AI Mode. Never advise blocking it.
robots.txt lines
User-agent: Googlebot Disallow: /
User-agent: Googlebot Allow: /
Questions
What is Googlebot?
Googlebot is Google's search crawler. It builds Google Search itself, which AI Overviews and AI Mode draw on.
Should I block Googlebot?
No. Blocking it removes the site from Google Search as well as from AI Overviews and AI Mode. Almost every business should allow it.
How do I block Googlebot in robots.txt?
To block Googlebot for the whole site, add these two lines to robots.txt: User-agent: Googlebot Disallow: /
Does blocking Googlebot affect AI search or Google?
Yes. Blocking Googlebot removes the site from Google Search as well as from AI Overviews and AI Mode.