llms.txt checker

check whether a domain publishes an llms.txt — a markdown index that gives AI agents a curated summary of the site's content. part of the domainposture.com domain dossier.

Overview

llms.txt is an emerging, not-yet-standardised convention (proposed by Jeremy Howard in 2024) that plays the same role for AI agents that robots.txt plays for crawlers and sitemap.xml plays for search engines: a single, predictable file at the domain root — `https://<domain>/llms.txt` — that a language model can fetch to get a curated markdown overview of the site instead of having to scrape and parse full HTML pages. the file convention is simple: it's plain markdown, starting with an H1 title, typically followed by a short summary and a linked list of the site's most important pages, often grouped into sections. there is no registration authority and no required schema beyond 'markdown starting with an H1' — different sites structure the body differently. this tool fetches the file, confirms it isn't a catch-all HTML response from a single-page app (a common false positive — a 200 response with HTML content for any path is not an llms.txt), and reports whether one is published along with its first line and size. presence is a content-governance signal, not a security control: it shows a site has deliberately curated what it wants AI agents to read, rather than leaving them to scrape whatever they find.

How to use

enter a bare domain — public fqdn only. no scheme, no port.
fetch llms.txt — the tool requests https://<domain>/llms.txt and checks the response is not HTML (a same-path catch-all from a single-page app is not an llms.txt) and starts with a markdown `#` heading.
read the summary — when present, the tool shows the first line (the H1 title) and the file size in bytes — enough to confirm the file is real without needing to open it separately.

Examples

example 1 — a real llms.txt starts with a markdown H1 and is plain text, not HTML.

input
site with a published llms.txt

output
firstLine: "# Acme Corp" · bytes: 2140

example 2 — a 200 response is not enough — the tool rejects HTML bodies to avoid false positives from SPA fallback routing.

input
single-page app answering every path with the same HTML shell

output
not_applicable — catch-all HTML response, not an llms.txt

Common mistakes

there is no official standard or validator — llms.txt is a community convention, not an RFC or W3C spec. there's no required schema beyond starting with an H1 in markdown — different sites structure the body (sections, linked pages, summaries) however they choose.
a 200 response doesn't guarantee it's real — many sites — especially single-page apps — answer every unmatched path with the same HTML shell instead of a 404. this tool specifically rejects text/html responses so a catch-all route doesn't get reported as a published llms.txt.
llms.txt is not the same as an AI-crawler policy — llms.txt is a positive content index — it tells agents what to read and how it's organised. it says nothing about whether GPTBot or ClaudeBot are allowed to crawl in the first place; that's governed separately by robots.txt (see the ai-crawler checker).
adoption is early and uneven — most domains don't publish one yet. absence is not a red flag in the way a missing security.txt can be — it's simply a maturity signal, not evidence of neglect.

FAQ

what is llms.txt?

a proposed convention — a plain markdown file at the domain root that gives AI agents a curated summary and link index of a site's most important content, so they don't have to scrape and parse full HTML pages.

is llms.txt an official web standard?

no. it was proposed by Jeremy Howard in 2024 and has been adopted informally by a growing number of sites, mostly documentation-heavy and developer-facing ones, but it isn't governed by any standards body.

how is llms.txt different from a sitemap.xml?

sitemap.xml is a structured XML list of URLs for search-engine crawlers, with no prose. llms.txt is free-form markdown intended to be read directly by a language model — it can include summaries, context, and curated groupings that a sitemap can't express.

does publishing llms.txt help my content get cited by AI answer engines?

it may make it easier for an agent to find and summarise your key pages accurately, but no major AI vendor has confirmed that llms.txt directly influences retrieval or citation ranking. treat it as a courtesy to AI readers, not a guaranteed ranking signal.

can ai agents call this?

yes — dossier_llms_txt on the mcp endpoint at domainposture.com/mcp/mcp.

what should an llms.txt file contain?

start with an H1 title naming the site, then a short summary paragraph, then a linked list of the most important pages — often grouped under H2 sections like 'docs' or 'api'. keep it concise; it's meant to be a map, not a full content mirror.

Related tools

ai-crawler checker — see whether a domain's robots.txt has declared a stance for GPTBot, ClaudeBot, and other major AI crawlers.
web surface inspector — fetch robots.txt, sitemap.xml, and the home page's <head> to summarise a domain's public-web surface.
security.txt checker — check whether a domain publishes a security.txt (RFC 9116) so researchers have a way to report vulnerabilities.

References

Free beta

Unlock your full scan (free beta)

Add one DNS TXT record to prove you own the domain, then we run a deeper active scan — exposed secrets and config — and send you a private report. Free beta, no payment.

Finds exposed .env / .git / backups and other sensitive files
Verified by a DNS TXT record, so only you can run it on your domain
A private report emailed straight to you

Unlock full scan