llms.txt: what it is and how to write one

llms.txt is a markdown file you publish at the root of your domain — https://example.com/llms.txt — that gives AI assistants a short, curated map of your site. When someone asks ChatGPT, Claude, or Perplexity about your product, the model can't read your whole site at inference time: context windows are small and rendered HTML is full of navigation, scripts, and ads. llms.txt hands the model a clean, hand-picked index of the pages that matter, each ideally linking to a plain-markdown version. It is guidance, not access control — it helps models use your content well, and it does nothing to block them.

What llms.txt is for

Search engines solved "find every page" with sitemap.xml. Language models have a different problem: given a small context budget, which few pages best answer a question about you, and where is the clean text? llms.txt answers exactly that. It is a curator's file — you decide what a model should read first, in priority order, and point at markdown rather than HTML so the model gets signal instead of noise.

That makes it most valuable for sites a model is likely to be asked about directly: documentation, developer tools, APIs, and products. If an assistant is going to explain your API or recommend your tool, llms.txt is your chance to make sure it reads the right, current pages.

The format

The spec is deliberately small. A valid file has:

• An # H1 with the site or project name (the only required line).
• An optional > blockquote with a one-line summary.
• Optional prose or ## sections, each containing a markdown list of links in the form [name](url): optional notes.
• An optional ## Optional section for links a model can skip if it's short on context.

A minimal example:

# Acme Docs
 
> Acme is a payments API for marketplaces. This file indexes the docs an assistant should read first.
 
## Docs
 
- [Quickstart](https://acme.dev/docs/quickstart.md): create your first charge in five minutes
- [Authentication](https://acme.dev/docs/auth.md): API keys, scopes, and rotation
- [Webhooks](https://acme.dev/docs/webhooks.md): event types and signature verification
 
## Optional
 
- [Changelog](https://acme.dev/changelog.md): release history

Point each link at a markdown version of the page where you can (.md), because that is what the model ingests best. For sites that want to hand over everything in one request, publish a companion llms-full.txt with the full expanded content inline.

llms.txt vs robots.txt vs sitemap.xml

These three files are easy to confuse — they live in similar places and all "talk to crawlers" — but they do different jobs:

• robots.txt controls access: which crawlers (including AI crawlers like GPTBot, ClaudeBot, Google-Extended) may fetch which paths. It is a fence.
• sitemap.xml is for search indexing: a machine-readable list of every URL you want crawled, aimed at Google and Bing.
• llms.txt is for LLM consumption: a short, curated, human-written guide to your best content, aimed at a model reading it at inference time. It is a recommendation, not a rule.

If your goal is to keep AI models out, llms.txt is the wrong file — set AI-crawler directives in robots.txt instead. llms.txt is for when you want to be quoted accurately.

Should you add one?

If AI assistants are plausibly asked about your product — docs, a dev tool, an API — yes. It is a few minutes of work, it can only help how models describe you, and it costs nothing if models ignore it. If you run a brochure site nobody asks an LLM about, it's optional.

Once you've published one, check that it actually parses and that its links resolve:

The catch

llms.txt is a proposal, not a ratified standard, and no major model vendor has publicly committed to reading it on every request. Treat it as low-cost insurance: cheap to publish, helpful where it's honoured, harmless where it isn't. The real risk is the opposite mistake — assuming llms.txt keeps models out. It doesn't. Access is robots.txt's job.