What robots.txt is, and where it lives
robots.txt is a plain-text file at the root of your domain — yoursite.com/robots.txt — that tells automated crawlers which parts of your site they may request. It implements the Robots Exclusion Protocol, first proposed in 1994 and finally standardized in 2022 as RFC 9309.
One rule frames everything else: it is voluntary. robots.txt is a set of instructions a crawler chooses to honor, not access control. Well-behaved bots read it first; a scraper that ignores it pays no penalty. It must live at the domain root (one file per host, per scheme and port) and return a 200 with a text/plain content type.
The core directives
- User-agent — opens a group and names the crawler it applies to. A literal * is the catch-all group for any bot without its own.
- Disallow — a path prefix the group may not fetch. Disallow: / blocks everything; an empty Disallow: blocks nothing.
- Allow — carves an exception back out of a broader Disallow (e.g. block /app/ but allow /app/public/).
- Sitemap — an absolute URL to your sitemap. Global, not tied to any group; you can list several.
- Crawl-delay — a non-standard hint (seconds between requests) honored by Bing and others but ignored by Google. Manage Googlebot’s rate in Search Console instead.
How matching and precedence work
This is the part most people get wrong. Under RFC 9309 a crawler selects the single most-specific group whose User-agent matches its own name — and then uses only that group. Rules from other groups, including the * group, are not merged in. If Googlebot has its own group, it obeys that group and ignores * entirely.
Within the chosen group, the most-specific path rule wins — specificity is measured by the length of the matched path. When an Allow and a Disallow match a URL with the same length, Allow wins. Order of lines does not matter; length does.
| Rules in group | URL | Result |
|---|---|---|
| Disallow: /app/ Allow: /app/public/ | /app/public/x | Allowed (longer match) |
| Disallow: /p Allow: /p | /page | Allowed (tie → Allow) |
Common recipes
- Allow everything — User-agent: * / Disallow: (an empty Disallow). The safe default.
- Block a path for all bots — User-agent: * / Disallow: /admin/.
- Block one specific bot — give it its own group: User-agent: BadBot / Disallow: /.
# https://yoursite.com/robots.txt
# Default rules for every crawler
User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /*?sessionid= # block session-id URLs (Google/Bing wildcard)
Allow: /
# Block one badly-behaved scraper entirely
User-agent: BadBot
Disallow: /
# Point every crawler at the sitemap (global, absolute URL)
Sitemap: https://yoursite.com/sitemap.xmlThe * and $ wildcards are Google/Bing extensions, not core RFC 9309 — reliable on the major search crawlers, but don’t assume every bot honors them.
AI crawlers in robots.txt
AI crawlers respect robots.txt the same way search crawlers do — you control them with a group naming the bot and Disallow/Allow rules. The tokens that matter today include GPTBot, ClaudeBot, Google-Extended, PerplexityBot, Applebot-Extended and CCBot. Deciding which to allow (retrieval bots that can cite you) versus block (training crawlers) is its own topic — the AI bot policies guide has copy-paste rules for every goal, and the AI bot checker shows what your current file allows.
Common mistakes
- ✕Blocking the whole site with Disallow: /. Usually a staging file that shipped to production. It quietly stops all crawling — the single most common way sites vanish from search.
- ✕Blocking CSS and JS. Disallowing /assets/ or /_next/ stops Google rendering the page as a user sees it, which can hurt how it’s evaluated. Let crawlers fetch the resources the page needs.
- ✕Treating it as security or enforcement. robots.txt is public and advisory. Never list secret paths in it (you’re publishing a map of them), and never rely on it to keep anyone out. Use auth for that.
- ✕Using it to deindex a page. Disallow blocks crawling, not indexing — a blocked page can still appear in results. To remove a page, allow crawling and use a noindex tag instead.
- ✕Serving it as HTML. If your framework returns the file wrapped in your app shell or with a text/html type, crawlers may misparse or ignore it. It must be plain text at a 200.
How to test it
After editing, fetch yoursite.com/robots.txt directly and confirm it returns a 200 as plain text — not your 404 page, not HTML. Then use Google Search Console’s robots.txt report to see how Googlebot parses each group and to test specific URLs against your rules.
AvocadoScore checks that your robots.txt exists, parses cleanly and doesn’t block crawlers or AI bots you meant to allow — as part of the readiness scorecard and the full audit.
Questions, answered
What is robots.txt?+
robots.txt is a plain-text file at the root of your domain (yoursite.com/robots.txt) that tells automated crawlers which parts of your site they may or may not request. It follows the Robots Exclusion Protocol, standardized in 2022 as RFC 9309. It is a set of instructions crawlers choose to honor — not a firewall, not access control. Well-behaved bots like Googlebot read it before crawling; it does not stop anyone who ignores it.
Does robots.txt block a page from Google?+
No — and this trips people up constantly. Disallow tells a crawler not to fetch a URL, but a disallowed page can still be indexed if other pages link to it, showing up in results without a snippet. To keep a page out of the index, let it be crawled and add a noindex meta tag (or an X-Robots-Tag header) — the crawler must be able to fetch the page to see the noindex. Blocking it in robots.txt hides the noindex, so it stays indexed.
What happens if I have no robots.txt at all?+
Crawlers treat a missing robots.txt (a 404) as "everything is allowed" and crawl freely. That is usually fine for a small site. A file that returns a 5xx server error is different: Google may treat repeated 5xx responses as "disallow everything" and pause crawling, so a broken robots.txt can be worse than none. Always serve it as a 200 with plain text.
How do wildcards and matching work in robots.txt?+
Under RFC 9309 a crawler picks the single most-specific group whose User-agent matches its name, and it uses only that group — rules from other groups (including the * group) are not merged in. Within a group, the most-specific path rule wins by length, and for a tie Allow beats Disallow. Google and Bing also support * as a wildcard and $ to anchor the end of a URL, though those extensions are not part of the core RFC.
Can robots.txt control AI crawlers like GPTBot?+
Yes, the same way it controls any crawler: with a User-agent group naming the bot (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, Applebot-Extended, CCBot) and Disallow/Allow rules. The major AI vendors document and honor these tokens. As with all robots.txt, it is voluntary compliance, not enforcement — see our AI bot policies guide for which bots to allow or block and why.
Where does the Sitemap directive go?+
Anywhere in the file, on its own line, as an absolute URL: Sitemap: https://yoursite.com/sitemap.xml. It is independent of any User-agent group and applies globally — you can list several. It is the standard way to point crawlers at your sitemap without submitting it in each search console.
Is your robots.txt helping or quietly hurting you?
AvocadoScore checks your robots.txt, sitemap, AI bot policies and the rest of your AI readiness across your site, with a ranked fix list. Free, every fix unlocked.