What llms.txt is
llms.txt is a plain-Markdown file you place at the root of your site — yoursite.com/llms.txt — that hands AI models a curated, human-readable map of your most important content. It was proposed by Jeremy Howard of Answer.AI in September 2024.
Think of it as a README written for large language models: a short description of what your site is, followed by curated links to the pages that matter most, grouped into sections. Instead of forcing a model to crawl your entire site and guess which pages are canonical, you tell it directly — in a format it parses effortlessly.
Why it exists
AI answer engines have a context problem. When someone asks ChatGPT or Perplexity about your product, the model has limited room and limited patience: it can’t ingest hundreds of pages, and JavaScript-heavy sites often render nearly empty to it. Left to guess, it may quote an outdated blog post, a pricing page from a cache, or nothing at all.
llms.txt is the fix on your side of the fence: a single, stable, text-first entry point that says “here is what I am, and here are the canonical pages to read.” It is cheap to publish, does no harm, and increasingly expected on developer-facing sites.
How it differs from robots.txt and sitemap.xml
| File | Answers | For |
|---|---|---|
| robots.txt | What may I access? | Permission |
| sitemap.xml | What URLs exist? | Coverage |
| llms.txt | What matters, and why? | Comprehension |
They are complementary. llms.txt doesn’t replace the other two — you still need robots.txt to allow AI crawlers (check yours with the AI bot checker) and sitemap.xml for full indexing.
The format
llms.txt is just Markdown with a light convention. The order and the headings matter — clients read them structurally:
- An H1 with your site or project name (required).
- A blockquote summary — one or two sentences on what the site is (strongly recommended).
- Zero or more H2 sections grouping curated links. Each link is [Title](url): short note.
- An optional ## Optional section at the end — links a model can skip if it’s short on context.
# Acme Analytics
> Acme Analytics is a privacy-first product analytics platform for SaaS
> teams. Self-hostable, EU-hosted, no cookies. Free up to 10k events/mo.
## Docs
- [Quickstart](https://acme.com/docs/quickstart): Install the SDK and send
your first event in under five minutes.
- [Event API](https://acme.com/docs/api/events): Reference for the /events
endpoint — payload shape, batching, rate limits.
- [Self-hosting](https://acme.com/docs/self-host): Deploy Acme on your own
infrastructure with Docker or Kubernetes.
## Product
- [Pricing](https://acme.com/pricing): Plans, limits, and the free tier.
- [Security](https://acme.com/security): Data residency, encryption, SOC 2.
## Optional
- [Changelog](https://acme.com/changelog): Release notes.
- [Blog](https://acme.com/blog): Long-form posts on analytics and privacy.The ## Optional section is a real convention, not a placeholder — it explicitly tells a model which links are safe to drop when context is tight.
llms.txt vs llms-full.txt
llms.txt is the concise index — summary plus curated links. llms-full.txt is optional and contains the actual expanded content, often your entire documentation concatenated into one file, so a model can ingest everything in a single request without following links. Publish llms.txt first. Add llms-full.txt only if you have a docs corpus worth serving whole.
Common mistakes
- ✕Serving it as HTML. If your framework returns the file wrapped in your app shell or with a text/html content type, clients treat it as absent. It must be plain text at a 200.
- ✕Dumping every URL. llms.txt is a curation, not a second sitemap. Twenty well-chosen links beat two thousand — the whole point is to signal what matters.
- ✕Skipping the summary. The blockquote is what a model reads first to decide what you are. Without it, the links have no framing.
- ✕Assuming it fixes access. A perfect llms.txt is useless if your robots.txt blocks AI bots or a CDN drops them at the edge. Comprehension can’t happen without access first.
- ✕Letting it rot. A stale llms.txt pointing at moved or dead pages is worse than none. Regenerate it when your site structure changes.
How to create one
You can write it by hand from the format above, or generate a starter automatically. Our free llms.txt generator walks your sitemap, groups your pages, and hands back a ready Markdown file you can edit and drop at your root. Then confirm the rest of your AI readiness — server-rendered content, structured data, crawler access — with the readiness checker.
Questions, answered
What is llms.txt?+
llms.txt is a plain-Markdown file placed at the root of your site (yoursite.com/llms.txt) that gives AI models a curated, human-readable map of your most important content. It was proposed by Jeremy Howard of Answer.AI in September 2024. Think of it as a README written for large language models: a short summary of what your site is, followed by curated links to the pages that matter, so an AI can find and understand your best content without crawling everything or guessing.
Is llms.txt an official standard?+
It is a proposed, community-driven convention rather than a ratified standard like robots.txt or sitemap.xml. No search engine or AI vendor is contractually obliged to read it. That said, it is cheap to add, does no harm, and a growing number of developer-facing sites publish one — so it is a low-risk, forward-looking signal, not a guaranteed ranking factor.
How is llms.txt different from robots.txt and sitemap.xml?+
robots.txt tells crawlers what they may access; sitemap.xml lists every URL for indexing; llms.txt curates and explains your best content for AI reasoning. robots.txt is permission, sitemap.xml is coverage, llms.txt is comprehension. They are complementary — llms.txt does not replace the other two.
What is the difference between llms.txt and llms-full.txt?+
llms.txt is the concise index: a summary plus curated links. llms-full.txt is optional and contains the actual expanded content (often your docs concatenated into one file) so an AI can ingest everything in a single request without following links. Publish llms.txt first; add llms-full.txt only if you have a documentation corpus worth serving whole.
Where do I put the llms.txt file?+
At the root of your domain, served as plain text at https://yoursite.com/llms.txt — the same location pattern as robots.txt. It must return a 200 with a text/plain (or Markdown) content type. If your framework serves it as HTML or 404s, AI clients treat it as absent.
Will llms.txt get my site cited by ChatGPT or Perplexity?+
Not on its own. Citation depends first on access (AI crawlers can reach your pages) and comprehension (your content is server-rendered, well-structured, and dated). llms.txt helps with comprehension by handing models a clean map, but it is one signal among many. Fix crawler access and server-rendered content first; add llms.txt as the finishing layer.
Is your whole site AI-ready — not just your llms.txt?
AvocadoScore scores access, comprehension and trust across your site, with a ranked fix list. Free, every fix unlocked.