Free tool · instant · no signup
Which AI crawlers can read your site?
Check whether ChatGPT, Claude, Gemini and Perplexity’s crawlers are allowed, blocked, or crawling by default — resolved straight from your robots.txt across a dozen major AI bots. Then a live GPTBot fetch proves whether your content is actually served, or quietly blocked by a CDN.
Two kinds of AI crawler, two different decisions
AI bots split into two camps, and blocking them means very different things. Live-retrieval bots — OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User — fetch your pages so answer engines can cite them in real time. Block these and you simply cannot appear in ChatGPT Search or Perplexity results. Training bots — GPTBot, Google-Extended, Applebot-Extended, CCBot — gather data to train models. Blocking those is a principled opt-out that does not affect whether you get cited live.
The common mistake is a blanket Disallow: / on User-agent: *, or copy-pasting a “block all AI” snippet, which quietly shuts out the retrieval bots that would have sent traffic. This checker shows each bot separately so you can make the call per crawler.
robots.txt is intent — not proof
Even a perfect robots.txt does not guarantee an AI bot can read you. A CDN or firewall can block bot-shaped requests at the edge, and a JavaScript-only page can render empty to crawlers that don’t run JS. That’s why this tool also fetches your homepage as GPTBot and reports how much of your content actually comes back. To go further — whether AI can parse and understand your pages, not just reach them — the full audit scores comprehension and trust across your whole site.
Questions, answered
What does this AI bot checker do?+
It reads your robots.txt and resolves, for every major AI crawler — GPTBot, OAI-SearchBot, ClaudeBot, Google-Extended, PerplexityBot, Applebot-Extended, CCBot and more — whether it is allowed, blocked, or crawling by default. Then it runs a live GPTBot fetch to prove whether your content is actually served, because a firewall or CDN can silently block a bot your robots.txt happily allows.
Why would I want AI bots to crawl my site?+
Live-retrieval crawlers (OAI-SearchBot, PerplexityBot, ChatGPT-User, Claude-User) decide whether your pages can be cited in ChatGPT Search, Perplexity and other answer engines. If you block them, you cannot be surfaced or quoted there. Training crawlers (GPTBot, Google-Extended, CCBot) are a separate choice — blocking them opts you out of model training without affecting live citation.
Should I block AI crawlers?+
It depends on your goal. If you want traffic and citations from AI answer engines, keep the live-retrieval bots allowed. If you want to opt out of model training on principle, block the training bots (GPTBot, Google-Extended, Applebot-Extended, CCBot) while leaving the retrieval bots open. Many sites get this backwards and accidentally block the bots that would have sent them traffic.
My robots.txt allows a bot but it still can’t read my site. Why?+
robots.txt is a stated policy — it is not enforcement. A CDN or WAF (Cloudflare, Akamai, a bot-management rule) can block a bot at the network edge before robots.txt is ever consulted, and JavaScript-only rendering can leave a bot a near-empty page. That is exactly why this tool also runs a live fetch as GPTBot and reports how much of your content it actually receives.
Is it free?+
Yes — the AI bot checker is free with no signup. For whether AI can actually parse and cite your pages (llms.txt, structured data, server-rendered content) across your whole site, plus Core Web Vitals and a ranked fix list, the full AvocadoScore audit is free too — no signup, every fix unlocked.