Reference

AI crawler directory

User agents, robots.txt rules and the real cost of blocking each crawler. Not sure where your site stands? The free AI bot checker tests them all at once. Looking for our own crawler? See AvocadoScoreBot.

• GPTBotOpenAI · ChatGPT and OpenAI model trainingAI training crawler
• OAI-SearchBotOpenAI · ChatGPT SearchAI search crawler
• ChatGPT-UserOpenAI · ChatGPT live browsingUser-triggered AI fetcher
• ClaudeBotAnthropic · Claude model trainingAI training crawler
• Claude-UserAnthropic · Claude live browsingUser-triggered AI fetcher
• Claude-SearchBotAnthropic · Claude search resultsAI search crawler
• PerplexityBotPerplexity · Perplexity answer engineAI search crawler
• Perplexity-UserPerplexity · Perplexity live fetchingUser-triggered AI fetcher
• Google-ExtendedGoogle · Gemini apps and Vertex AIAI training crawler
• Applebot-ExtendedApple · Apple Intelligence model trainingAI training crawler
• CCBotCommon Crawl · Common Crawl open web corpusAI training crawler
• BytespiderByteDance · ByteDance AI productsAI training crawler
• AmazonbotAmazon · Alexa and Amazon AI servicesAI search crawler
• Meta-ExternalAgentMeta · Meta AI and Llama trainingAI training crawler
• SiteCheckerBotCrawlerSitechecker · Sitechecker SEO auditsSEO audit crawler