Crawler directory · AI training crawler
Applebot-Extended: user agent, robots.txt rules, and how to allow or block it
Applebot-Extended is a robots.txt token that controls whether content Applebot has crawled may be used to train Apple’s foundation models. It never fetches pages itself, and disallowing it does not stop normal Applebot from indexing you for Siri and Spotlight.
- Operator
- Apple
- Feeds
- Apple Intelligence model training
- Type
- AI training crawler
- robots.txt token
- Applebot-Extended
- Fetches pages itself
- No — it is a permission token only
Applebot-Extended user agent
Applebot-Extended never makes requests, so it has no User-Agent header of its own — it exists only as a robots.txt token. The requests you see in your logs come from the vendor’s normal crawlers.
Official documentation: support.apple.com. Vendors change their crawlers, so check it for the current details.
Should you allow or block Applebot-Extended?
Blocking stops future crawls from being used for model training. It does not remove content already collected, and it does not stop you being cited in live answers, which use separate retrieval bots.
robots.txt rules for Applebot-Extended
Put these in the robots.txt at the root of your domain.
Block Applebot-Extended
User-agent: Applebot-Extended Disallow: /
Explicitly allow Applebot-Extended
User-agent: Applebot-Extended Allow: /
Block only part of your site
User-agent: Applebot-Extended Disallow: /private/ Allow: /
robots.txt states your policy; it does not enforce it. A CDN or firewall can block a bot your robots.txt allows, which is why it is worth testing rather than assuming.
Check what your site does today
The free AI bot checker reads your robots.txt and tells you whether GPTBot, ClaudeBot, PerplexityBot and nine other AI crawlers are allowed, blocked or crawling by default, then runs a live GPTBot fetch to prove your content is actually served.
Check my site free →Questions, answered
What is Applebot-Extended?+
Applebot-Extended is a robots.txt token that controls whether content Applebot has crawled may be used to train Apple’s foundation models. It never fetches pages itself, and disallowing it does not stop normal Applebot from indexing you for Siri and Spotlight.
How do I block Applebot-Extended in robots.txt?+
Add these two lines to the robots.txt at the root of your domain: User-agent: Applebot-Extended Disallow: / User-agent tokens are matched case-insensitively. The rule applies from the crawler's next visit; it does not delete anything already collected.
Will blocking Applebot-Extended affect my Google rankings?+
No. Applebot-Extended is separate from Googlebot, so blocking it does not change how Google Search indexes or ranks you. Blocking stops future crawls from being used for model training. It does not remove content already collected, and it does not stop you being cited in live answers, which use separate retrieval bots.
Related crawlers
More on the trade-offs in the AI bot policies guide and the robots.txt guide.