Crawler directory · AI training crawler
Bytespider: user agent, robots.txt rules, and how to allow or block it
Bytespider is ByteDance’s crawler, used to gather data for its AI products. Site operators frequently report it as high-volume.
- Operator
- ByteDance
- Feeds
- ByteDance AI products
- Type
- AI training crawler
- robots.txt token
- Bytespider
Bytespider user agent
Requests from Bytespider carry a User-Agent header like this (example — vendors change version numbers, so match on the token Bytespider, not the whole string):
Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; spider-feedback@bytedance.com)
To see how often it visits, filter your access log with grep -i "Bytespider" access.log. A User-Agent can be faked, so treat it as a claim, not proof.
Should you allow or block Bytespider?
Blocking stops future crawls from being used for model training. It does not remove content already collected, and it does not stop you being cited in live answers, which use separate retrieval bots.
Heads up: Some operators report that robots.txt alone does not always hold it back, so pair the rule with a server-level block if load is a concern.
robots.txt rules for Bytespider
Put these in the robots.txt at the root of your domain.
Block Bytespider
User-agent: Bytespider Disallow: /
Explicitly allow Bytespider
User-agent: Bytespider Allow: /
Block only part of your site
User-agent: Bytespider Disallow: /private/ Allow: /
robots.txt states your policy; it does not enforce it. A CDN or firewall can block a bot your robots.txt allows, which is why it is worth testing rather than assuming.
Check what your site does today
The free AI bot checker reads your robots.txt and tells you whether GPTBot, ClaudeBot, PerplexityBot and nine other AI crawlers are allowed, blocked or crawling by default, then runs a live GPTBot fetch to prove your content is actually served.
Check my site free →Questions, answered
What is Bytespider?+
Bytespider is ByteDance’s crawler, used to gather data for its AI products. Site operators frequently report it as high-volume.
How do I block Bytespider in robots.txt?+
Add these two lines to the robots.txt at the root of your domain: User-agent: Bytespider Disallow: / User-agent tokens are matched case-insensitively. The rule applies from the crawler's next visit; it does not delete anything already collected. Some operators report that robots.txt alone does not always hold it back, so pair the rule with a server-level block if load is a concern.
Will blocking Bytespider affect my Google rankings?+
No. Bytespider is separate from Googlebot, so blocking it does not change how Google Search indexes or ranks you. Blocking stops future crawls from being used for model training. It does not remove content already collected, and it does not stop you being cited in live answers, which use separate retrieval bots.
What does the Bytespider user agent look like?+
An example User-Agent header is: Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; spider-feedback@bytedance.com). Vendors change version numbers, so match on the token Bytespider rather than the whole string. A User-Agent can be spoofed, so verify important traffic against the vendor's published details before trusting it.
Related crawlers
More on the trade-offs in the AI bot policies guide and the robots.txt guide.