What Is llms.txt? Format, Real Adoption Data, and How to Write One

llms.txt is a proposed text file, placed at a site's root like /llms.txt, that gives AI models a short, curated map of a site's most important pages in Markdown. Jeremy Howard, co-founder of Answer.AI and fast.ai, first published the proposal in September 2024. As of mid-2026 it has a defined format and growing publisher adoption, but no major AI crawler (OpenAI's GPTBot, Anthropic's Claude, Google's Gemini/Google-Extended, or Perplexity) has confirmed it actually reads the file when crawling other people's sites. This guide covers the format, why it exists, what real usage data shows, and how to write one if you decide it's worth your time.

What is llms.txt?

llms.txt is a Markdown file hosted at /llms.txt (or a subpath, like /docs/llms.txt) that summarizes a site or section of it for an AI agent: what the site is, and links to the pages that matter most, each with a one-line description. The idea mirrors robots.txt and sitemap.xml: a standard filename at a predictable location, so any agent can check for it without needing to be told where to look.

The problem it targets is real even if the fix is unproven. A web page is built for a human eye: navigation bars, ads, cookie banners, and JavaScript wrap the actual content. An AI agent trying to answer a question or complete a task has to strip all of that out just to get to the text, and it's burning context window tokens doing it. llms.txt proposes handing the agent a clean, pre-summarized starting point instead.

Unlike a general-purpose instructions file such as AGENTS.md, which is freeform prose telling a coding agent how to work in a repo, llms.txt follows a specific structure. It isn't a place to write open-ended notes. The spec (maintained at llmstxt.org) defines an exact section order, and a compliant parser can process it with regular code, without needing an LLM to read it loosely.

The format, exactly

Per the official spec, a valid llms.txt file contains these sections in this order:

• An optional byte-order mark (BOM)
• An H1 with the project or site name. This is the only required section
• A blockquote directly under the H1, with a short summary of the site
• Zero or more plain Markdown sections (paragraphs, lists) with more context, but no headings
• Zero or more H2-delimited sections, each a list of links in [name](url): optional notes format

Here's the minimal structure from the spec itself:

```markdown

Title

Optional description goes here

Optional details go here

Section name

• Link title: Optional link details

Optional

• Link title ```

The "Optional" section name is a spec convention: it marks links an agent can skip if it needs a shorter context window. Anthropic's own published file (at docs.anthropic.com/llms.txt) and OpenAI's developer docs file (developers.openai.com/llms.txt) both follow this structure, though the spec explicitly allows sites to interpret the level of detail differently. Semrush's review of live examples from Hugging Face, Vercel, Zapier, and Cal.com found each company structuring the body differently, some heavy on subheadings and code blocks, others a flat list of links, while all stayed within the required H1-then-blockquote-then-H2-lists skeleton.

The spec also proposes a companion convention: serving a clean Markdown version of any HTML page at the same URL with .md appended (/docs/page.html.md), so the file an llms.txt links to is itself easy for a model to parse.

Why it exists

Jeremy Howard framed the problem plainly in the v2 spec: agents fetch websites constantly now, whether that's a coding agent pulling API docs or a chat assistant reading a product page, and HTML wasn't built for that. Context windows are bigger than they were in 2024, but a full site still doesn't fit, and every token spent parsing navigation chrome is a token not spent on the actual question.

llms.txt doesn't replace robots.txt or sitemap.xml, and the spec says so directly. robots.txt controls crawl permission. sitemap.xml lists every indexable page for search engines, more pages than would ever fit in a model's context and without the curation an agent needs. llms.txt sits in a different slot: a short, curated pointer to the handful of pages that actually matter, meant to be read on demand when an agent needs specific information, not during a training crawl.

The honest state of adoption: publishers are ahead of consumers

This is the part worth getting right before you spend a developer's afternoon on it: writing an llms.txt file and having it actually change what an AI system does are two different things, and right now only the first one is happening at scale.

Publishing side, real and measured. Ahrefs analyzed 137,000 domains in its Web Analytics data (published June 2026) and found 28% already publish an llms.txt file. Mintlify generates one automatically for every site it hosts. GitBook, Wix, Yoast SEO, and AIOSEO all ship generator support. That's genuine, growing adoption on the publishing side, and it's the reason this convention is worth knowing about at all.

Consumption side, still unconfirmed. No major AI lab has stated that its crawler reads or prioritizes llms.txt when visiting an arbitrary third-party site. Ahrefs put it plainly: "no major LLM provider currently supports llms.txt. Not OpenAI. Not Anthropic. Not Google." Google folded a reference to llms.txt into its Agent2Agent protocol in 2025, which is a proposed protocol referencing a proposed file, not a crawling commitment. Anthropic and OpenAI both publish their own llms.txt files for their developer docs, which shows they're comfortable with the format as authors, not that ClaudeBot or GPTBot go looking for one on someone else's domain.

The clearest data point is server-log evidence, not speculation. Semrush ran a real test: it added an llms.txt file to its own property, Search Engine Land, in March 2025 and checked server logs from mid-August to late October 2025. Google-Extended, GPTBot, PerplexityBot, and ClaudeBot all recorded zero requests for the file in that window. Traditional crawlers (Googlebot, Bingbot) visited it a handful of times, with no sign of special treatment. Ahrefs found the same pattern at larger scale: of roughly 38,000 domains in its dataset with a valid llms.txt, 97% received zero requests for it in May 2026.

Google's John Mueller has been the most direct public voice on this. He's compared llms.txt to the old keywords meta tag: a claim a site owner makes about their own content that no search engine ever had to trust, because it can just check the content directly. Asked about the apparent contradiction between Google's own generative-search guidance (which says structured files like llms.txt "aren't needed" to appear in AI answers) and Chrome shipping an llms.txt check in Lighthouse, Mueller called the file "not done for search," describing it as, at most, a token-saving convenience for coding agents parsing developer docs.

That Lighthouse check is real, and it's the strongest institutional signal in llms.txt's favor: Chrome's agentic-browsing audits do flag a missing llms.txt file, framed as helping agents avoid crawling a full site to find its structure. But an audit that checks whether the file exists is not the same claim as an AI crawler confirming it fetches and uses that file's contents, and neither Google nor any other lab has made that second claim publicly.

The practical read: write one if it's low-effort and you already have clean, structured content to link to. Don't expect it to move AI citations or traffic on its own, and don't let anyone sell you an "llms.txt audit" as if it were a proven ranking factor. It isn't, not yet, and the people building the crawlers are the ones saying so.

How llms.txt relates to robots.txt and sitemap.xml

The three files aren't competitors. robots.txt decides who's allowed in. sitemap.xml lists everything that exists. llms.txt, if a given AI system chooses to read it, is a shortcut past both, straight to a curated summary. None of them substitute for the others.

How to write an llms.txt file

If you decide the low effort is worth trying, here's the sequence:

1. Pick the pages that actually matter. Product or service pages, core documentation, pricing, and your most current, evergreen guides. Skip old blog posts and anything you wouldn't want quoted out of context.
2. Write the H1 and blockquote first. One line naming the site, one blockquote sentence summarizing what it is. This is the only part of the file some agents may read in full.
3. Group links under H2 sections. Use plain nouns for section names (Docs, Products, Guides), not marketing language.
4. Write real descriptions, not titles restated. [API Reference](url): Authentication, rate limits, and endpoint list tells an agent something. [API Reference](url): Our API reference doesn't.
5. Keep an Optional section for anything skippable. Per the spec convention, put secondary links here so an agent working with a smaller context budget can ignore them.
6. Host it at the root as plain text, yourdomain.com/llms.txt, served with a text/markdown or text/plain content type.
7. Skip the exotic extras. Some sites have started adding custom metadata like token budgets to their files; there's no evidence any crawler reads or respects fields outside the core spec, so don't spend time inventing your own.
8. Update it when your key pages change. A stale llms.txt pointing at a deprecated docs page is worse than no file at all.

None of this requires a developer sprint. A single Markdown file, reviewed and versioned like any other content asset, is the whole deliverable.

FAQ

Does llms.txt help with SEO or Google rankings? No. It's not a ranking signal, and Google's own generative-search guidance says machine-readable files like this "aren't needed" to appear in AI-generated answers. Don't confuse it with technical SEO fundamentals like a clean sitemap or correct robots.txt directives, which do have confirmed effects on crawling and indexing.

Do ChatGPT, Claude, or Gemini actually read my llms.txt file? Not confirmed. As of mid-2026, no major AI lab has stated that its crawler checks for or prioritizes llms.txt on third-party sites. Independent server-log testing by both Ahrefs and Semrush found essentially zero requests for the file from GPTBot, ClaudeBot, PerplexityBot, or Google-Extended.

Is llms.txt the same thing as AGENTS.md? No. AGENTS.md is freeform prose for coding agents working inside a repository: build commands, conventions, project quirks. llms.txt follows a fixed section order (H1, then blockquote, then H2-delimited link lists) and is meant to be parsed by simple, non-AI tooling as well as read by a model.

Should I still create one? If you already have clean, well-organized documentation or product pages, writing an llms.txt file costs little and creates no real downside. Treat it as a low-priority, low-risk addition, not a project you need to prioritize over content structure, schema markup, or the technical SEO basics that have a confirmed effect on both traditional and AI search.

Where do I check the current spec? The authoritative source is llmstxt.org, maintained by Answer.AI. The format has already changed once (v1 to v2), so check there rather than relying on a third-party summary, including this one, for the exact current syntax.