Crawler directory · AI search crawler
PerplexityBot: user agent, robots.txt rules, and how to allow or block it
PerplexityBot builds the index behind Perplexity’s answers and citations. Perplexity says it is not used to train foundation models, so blocking it costs you visibility, not training opt-out.
- Operator
- Perplexity
- Feeds
- Perplexity answer engine
- Type
- AI search crawler
- robots.txt token
- PerplexityBot
PerplexityBot user agent
Requests from PerplexityBot carry a User-Agent header like this (example — vendors change version numbers, so match on the token PerplexityBot, not the whole string):
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
To see how often it visits, filter your access log with grep -i "PerplexityBot" access.log. A User-Agent can be faked, so treat it as a claim, not proof.
Official documentation: docs.perplexity.ai. Vendors change their crawlers, so check it for the current details.
Should you allow or block PerplexityBot?
Blocking removes your pages from that product’s index, so you are less likely to be surfaced or cited in its answers. Block it only if you do not want that traffic.
robots.txt rules for PerplexityBot
Put these in the robots.txt at the root of your domain.
Block PerplexityBot
User-agent: PerplexityBot Disallow: /
Explicitly allow PerplexityBot
User-agent: PerplexityBot Allow: /
Block only part of your site
User-agent: PerplexityBot Disallow: /private/ Allow: /
robots.txt states your policy; it does not enforce it. A CDN or firewall can block a bot your robots.txt allows, which is why it is worth testing rather than assuming.
Check what your site does today
The free AI bot checker reads your robots.txt and tells you whether GPTBot, ClaudeBot, PerplexityBot and nine other AI crawlers are allowed, blocked or crawling by default, then runs a live GPTBot fetch to prove your content is actually served.
Check my site free →Questions, answered
What is PerplexityBot?+
PerplexityBot builds the index behind Perplexity’s answers and citations. Perplexity says it is not used to train foundation models, so blocking it costs you visibility, not training opt-out.
How do I block PerplexityBot in robots.txt?+
Add these two lines to the robots.txt at the root of your domain: User-agent: PerplexityBot Disallow: / User-agent tokens are matched case-insensitively. The rule applies from the crawler's next visit; it does not delete anything already collected.
Will blocking PerplexityBot affect my Google rankings?+
No. PerplexityBot is separate from Googlebot, so blocking it does not change how Google Search indexes or ranks you. Blocking removes your pages from that product’s index, so you are less likely to be surfaced or cited in its answers. Block it only if you do not want that traffic.
What does the PerplexityBot user agent look like?+
An example User-Agent header is: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot). Vendors change version numbers, so match on the token PerplexityBot rather than the whole string. A User-Agent can be spoofed, so verify important traffic against the vendor's published details before trusting it.
Related crawlers
More on the trade-offs in the AI bot policies guide and the robots.txt guide.