Crawler directory · AI search crawler
Claude-SearchBot: user agent, robots.txt rules, and how to allow or block it
Claude-SearchBot indexes web content to improve the quality and relevance of Claude’s search results. Block it and your pages are less likely to be surfaced there.
- Operator
- Anthropic
- Feeds
- Claude search results
- Type
- AI search crawler
- robots.txt token
- Claude-SearchBot
Claude-SearchBot user agent
Match on the token Claude-SearchBot (case-insensitive) in the User-Agent header. In your access logs, filter with grep -i "Claude-SearchBot" access.log.
Official documentation: support.anthropic.com. Vendors change their crawlers, so check it for the current details.
Should you allow or block Claude-SearchBot?
Blocking removes your pages from that product’s index, so you are less likely to be surfaced or cited in its answers. Block it only if you do not want that traffic.
robots.txt rules for Claude-SearchBot
Put these in the robots.txt at the root of your domain.
Block Claude-SearchBot
User-agent: Claude-SearchBot Disallow: /
Explicitly allow Claude-SearchBot
User-agent: Claude-SearchBot Allow: /
Block only part of your site
User-agent: Claude-SearchBot Disallow: /private/ Allow: /
robots.txt states your policy; it does not enforce it. A CDN or firewall can block a bot your robots.txt allows, which is why it is worth testing rather than assuming.
Check what your site does today
The free AI bot checker reads your robots.txt and tells you whether GPTBot, ClaudeBot, PerplexityBot and nine other AI crawlers are allowed, blocked or crawling by default, then runs a live GPTBot fetch to prove your content is actually served.
Check my site free →Questions, answered
What is Claude-SearchBot?+
Claude-SearchBot indexes web content to improve the quality and relevance of Claude’s search results. Block it and your pages are less likely to be surfaced there.
How do I block Claude-SearchBot in robots.txt?+
Add these two lines to the robots.txt at the root of your domain: User-agent: Claude-SearchBot Disallow: / User-agent tokens are matched case-insensitively. The rule applies from the crawler's next visit; it does not delete anything already collected.
Will blocking Claude-SearchBot affect my Google rankings?+
No. Claude-SearchBot is separate from Googlebot, so blocking it does not change how Google Search indexes or ranks you. Blocking removes your pages from that product’s index, so you are less likely to be surfaced or cited in its answers. Block it only if you do not want that traffic.
Related crawlers
More on the trade-offs in the AI bot policies guide and the robots.txt guide.