The one distinction that matters
AI crawlers fall into two groups, and treating them the same is the mistake behind most bad policies.
Retrieval bots — OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User — fetch your pages live so an answer engine can cite you right now. Block these and you disappear from ChatGPT Search, Perplexity and similar.
Training bots — GPTBot, Google-Extended, ClaudeBot, CCBot, Applebot-Extended — gather data to train future models. Blocking them is a principled opt-out that does not affect whether you get cited today.
The classic error is a blanket “block all AI” snippet that shuts out the retrieval bots which would have sent you traffic. Decide per group, not with one switch.
Pick your policy — copy, paste, deploy
Add one of these to your robots.txt. They are additive — you can drop a block into an otherwise-open file.
# Maximum AI visibility — every answer engine can read and cite you.
# (This is also the default if you name no AI bots at all.)
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /# Allow live citation, opt out of model training.
# Retrieval bots (get you cited) — allowed:
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-User
Allow: /
# Training bots — blocked:
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
# Note: Google-Extended governs BOTH Gemini training AND AI Overviews.
# Blocking it opts you out of AI Overviews citation too. Leave it Allowed
# if you want to appear there:
User-agent: Google-Extended
Allow: /# Opt out of AI entirely (training and live retrieval).
# You will not appear in ChatGPT Search, Perplexity, or AI Overviews.
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-User
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: CCBot
Disallow: /robots.txt is intent — not enforcement
Reputable crawlers (OpenAI, Anthropic, Google, Perplexity, Apple) document that they respect robots.txt. But it is a voluntary standard: some bots ignore it, and — more commonly — a CDN or WAF can block a bot at the network edge before robots.txt is ever read, so a bot you meant to allow gets a challenge page instead. A JavaScript-only page can also render empty to crawlers that don’t run JS.
That is why you should always verify the real outcome after deploying. Our free AI bot checker reads your robots.txt policy for every major crawler and runs a live GPTBot fetch to prove whether your content is actually served.
Common mistakes
- ✕Blocking retrieval to stop training. Disallowing GPTBot does not stop ChatGPT citations — that is OAI-SearchBot and ChatGPT-User. Block the training bot; keep the retrieval bots open.
- ✕A blanket Disallow: / on User-agent: *. That blocks Google’s normal search crawler too, not just AI — almost always unintended.
- ✕Forgetting Google-Extended is dual-purpose. It controls Gemini training and AI Overviews. Block it only if you accept losing the AI Overviews surface.
- ✕Trusting the file without verifying. A CDN rule or an HTML-served robots.txt can quietly override your intent. Always confirm with a live check.
Questions, answered
Should I allow or block AI crawlers?+
It depends on your goal. If you want traffic and citations from AI answer engines (ChatGPT Search, Perplexity, Google AI Overviews), keep the live-retrieval bots allowed — blocking them removes you from those answers. Blocking the training bots (GPTBot, Google-Extended, CCBot, Applebot-Extended) is a separate, principled choice that opts you out of model training without affecting live citation, with one caveat: Google-Extended also governs AI Overviews, so blocking it costs you that surface too.
What is the difference between a training bot and a retrieval bot?+
Training bots (GPTBot, Google-Extended, ClaudeBot, CCBot, Applebot-Extended) gather data to train models. Retrieval bots (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) fetch your pages live so an answer engine can cite them in real time. Blocking a training bot affects future models; blocking a retrieval bot affects whether you can be cited today.
Does blocking GPTBot stop ChatGPT from citing me?+
No — those are different crawlers. GPTBot is OpenAI’s training crawler. ChatGPT’s live citations come from OAI-SearchBot (its search index) and ChatGPT-User (live browsing). You can block GPTBot to opt out of training while still allowing OAI-SearchBot and ChatGPT-User so ChatGPT can find and cite you.
Is robots.txt legally binding for AI crawlers?+
robots.txt is a voluntary standard, not a legal contract. The major, reputable AI crawlers (OpenAI, Anthropic, Google, Perplexity, Apple) publicly document that they respect it. Some crawlers ignore it. robots.txt is the right first tool for stating intent, but it is not enforcement — a CDN/WAF rule is what actually blocks a bot that refuses to comply.
Where do these rules go?+
In your robots.txt file at the root of your domain (yoursite.com/robots.txt), served as plain text. Each User-agent line names a bot; the Disallow / Allow lines beneath it are that bot’s rules. After you deploy, verify the result with a live check rather than trusting the file alone.
Check your live AI bot policy in seconds
See which crawlers your robots.txt actually allows, plus a live GPTBot read test — free, no signup.