At a glance
| robots.txt name | Bytespider |
|---|---|
| Company | ByteDance |
| Kind | Model training. Collects pages to train ByteDance's models |
| Visits sites | Yes |
Bytespider is a training crawler, meaning it collects pages that may be used to train AI models. It is owned by ByteDance and visits websites to gather data. Blocking it is a common and reasonable choice for many businesses.
Whether to block or allow Bytespider depends on your business goals. If you do not want your content used for training ByteDance's models, you should block it. Blocking Bytespider does not stop AI search crawlers, so it will not affect how AI assistants recommend your site. To block it, add a group to robots.txt with its User-agent line and Disallow: /. To allow it, use Allow: / or no rule for it.
robots.txt lines
User-agent: Bytespider Disallow: /
User-agent: Bytespider Allow: /
Questions
What is Bytespider?
Bytespider is an AI crawler from ByteDance that collects pages to train ByteDance's models. It visits sites to gather content for model training.
Should I block Bytespider?
Blocking Bytespider is a common, reasonable choice if you do not want your content used for training ByteDance's models. It does not stop AI search crawlers, so it will not affect how AI assistants recommend your site.
How do I block Bytespider in robots.txt?
To block Bytespider for the whole site, add these two lines to robots.txt: User-agent: Bytespider Disallow: /
Does blocking Bytespider affect AI search or Google?
No. Blocking Bytespider does not stop AI search crawlers, so it will not affect how AI assistants recommend your site. It also does not affect Google.