llms.txt

llms.txt is a plain-Markdown file, conventionally served at https://example.com/llms.txt, that gives a large language model a short, structured summary of a site or product and links to the pages worth reading in more depth. It was proposed in September 2024 by Jeremy Howard of Answer.AI as a community convention, not a formal web standard, and its canonical specification lives at llmstxt.org. The idea borrows the placement pattern of robots.txt and sitemap.xml, a well-known file at a predictable URL, but solves a different problem: not what a crawler is allowed to fetch, but what a language model should read first to understand a site efficiently.

The Problem It Solves

An LLM’s context window is finite, and even a generously sized one is small relative to a real documentation site or product catalog. Feeding a model raw HTML compounds the problem: a typical page carries navigation chrome, ad markup, cookie banners, and JavaScript bundles alongside the actual content, so a large fraction of every token spent “reading” the page is wasted on boilerplate rather than substance. Converting that HTML into something an LLM can use precisely, stripped of noise, structured, unambiguous, is also nontrivial to do well at request time.

llms.txt sidesteps both problems by asking site owners to hand-curate the map instead of asking an agent to infer it. The file itself is Markdown, deliberately, because Markdown is close to the format LLMs were trained on and requires no bespoke parser.

Anatomy of the Spec

The spec (v2, updated August 2026) defines an ordered structure. Only the H1 is strictly required; everything else is optional but conventionally present:

graph TD
    A["# Project Title (H1, required)"] --> B["> One-paragraph summary (blockquote)"]
    B --> C["Freeform context paragraphs"]
    C --> D["## Docs (H2 section)"]
    C --> E["## Examples (H2 section)"]
    C --> F["## Optional (H2, skippable)"]
    D --> G["- Link: description"]
    E --> H["- Link: description"]
    F --> I["- Link: description"]

    classDef default fill:#ffffff,stroke:#4338CA,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef data    fill:#EEF0F7,stroke:#0D9488,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef process fill:#F7F8FC,stroke:#6366F1,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    class A,B process;
    class C,D,E,F default;
    class G,H,I data;

The ## Optional convention matters in practice: an agent operating under a tight token budget can read the H1, blockquote, and primary sections while dropping the Optional block entirely, without needing to understand the content of those links to know it’s safe to skip them.

llms.txt vs. robots.txt vs. sitemap.xml

llms.txt sits alongside two much older well-known files at a site’s root, and it’s worth being precise about how it differs from both, since the naming pattern invites confusion:

The key difference is intent, not mechanism. robots.txt says what’s off-limits; sitemap.xml says what exists; llms.txt says what matters most and hands it over pre-cleaned. None of the three files are mutually exclusive, a well-run docs site typically ships all three, each solving a different part of the discoverability problem.

What Makes a Good llms.txt

The spec is permissive by design, which means quality varies enormously in practice. A weak file just dumps every URL from the sitemap into one flat list with no descriptions, offering little over the sitemap it duplicates. A strong file does the curation work a crawler can’t do for itself:

• The blockquote summary answers “what is this and why would an agent be here” in one sentence, not marketing copy.
• Sections are organized by task (“Getting started,” “Authentication,” “Migration guides”) rather than by internal site structure.
• Each link carries a short description of what’s actually on that page, not just its title, so an agent can decide relevance without fetching it.
• Low-priority material (changelogs, legal pages, brand assets) goes under ## Optional rather than competing for attention with the docs an agent actually needs.
• Linked pages are themselves clean Markdown (often the same .md source that renders the HTML page) rather than links back into full HTML, so the token savings shown below actually materialize once the agent follows a link.

llms.txt vs. Raw HTML: Where the Token Savings Come From

The core trade-off llms.txt exploits is token density: the same context budget buys far more usable content when it’s clean Markdown instead of noisy HTML.

At small budgets the gap is proportional; the same 32,000-token budget that fits roughly two raw HTML pages fits over a dozen curated Markdown pages. The ## Optional convention compounds this further by letting an agent shed low-priority links before it even fetches anything.

Writing and Generating One

A small site can write llms.txt by hand, as in the frontmatter example above. Larger docs sites generate it from existing sitemaps or documentation builds:

# Generate a minimal llms.txt from a docs site's page metadata
import yaml

def build_llms_txt(project_name: str, summary: str, pages: list[dict]) -> str:
    lines = [f"# {project_name}", "", f"> {summary}", ""]
    sections: dict[str, list[dict]] = {}
    for page in pages:
        sections.setdefault(page["section"], []).append(page)

    for section, section_pages in sections.items():
        lines.append(f"## {section}")
        for page in section_pages:
            desc = f": {page['description']}" if page.get("description") else ""
            lines.append(f"- [{page['title']}]({page['url']}){desc}")
        lines.append("")

    return "\n".join(lines)

pages = yaml.safe_load(open("docs_manifest.yaml"))
open("public/llms.txt", "w").write(
    build_llms_txt("Acme SDK", "Typed API client for Acme.", pages)
)

Documentation platforms increasingly automate this step entirely: Mintlify, Fern, GitBook, and Vercel’s docs framework now generate llms.txt for hosted documentation without the site owner writing it by hand.

Where It Fits in the Agent / RAG Stack

llms.txt is not a retrieval system by itself, it is a discovery layer that sits in front of one. An agent (a coding assistant, a browsing agent, a RAG pipeline’s ingestion step) fetches the file once, reads the curated map, and decides which linked pages are worth pulling in full:

graph LR
    A["Agent needs info about a product/API"] --> B["Fetch /llms.txt"]
    B --> C{"Relevant section found?"}
    C -->|"Yes"| D["Fetch only the 1-3 linked pages that matter"]
    C -->|"No, need more"| E["Fall back to full crawl or search"]
    D --> F["Feed clean Markdown into context / RAG index"]
    E --> F

    classDef default fill:#ffffff,stroke:#4338CA,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef data    fill:#EEF0F7,stroke:#0D9488,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef process fill:#F7F8FC,stroke:#6366F1,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    class A,F data;
    class B,D,E process;
    class C default;

This is the same pattern that makes Model Context Protocol servers useful: a small, structured entry point that lets an agent decide what to fetch next, instead of forcing it to ingest everything up front. In practice, coding agents and IDE tools (Cursor, Windsurf, Claude Code, GitHub Copilot, Cline, Aider) look for /llms.txt and a companion /llms-full.txt, a full-text concatenation of the same docs used by some implementers as a single-file alternative when an agent wants everything in one fetch rather than following individual links, when a user points them at a documentation URL.

What’s New (2025-2026)

Adoption stayed niche through most of 2025. It accelerated sharply after Mintlify rolled out automatic /llms.txt generation across every docs site it hosts, which meant thousands of existing documentation sites, including Anthropic’s and Cursor’s, gained an llms.txt overnight without their teams doing anything. By 2026, companies including Anthropic, Cloudflare, Vercel, Stripe, Supabase, and LangGraph publish one; industry estimates put overall adoption around 10% of sites. Anthropic’s own API documentation ships both a compact llms.txt (roughly 8,000 tokens) and a llms-full.txt (roughly 480,000 tokens) side by side, illustrating the curated-map-versus-everything trade-off at real scale. The spec itself saw a v2 revision in August 2026.

A Real-World Example: Anthropic’s Two Files

Anthropic’s own developer documentation is a useful case study in the curated-vs-complete trade-off the earlier chart illustrates, because it ships both extremes side by side rather than picking one. The compact llms.txt (roughly 8,000 tokens) reads like a table of contents: product names, one-line descriptions, and links into the docs, cheap enough that an agent can read the whole thing before deciding what to fetch next. The companion llms-full.txt (roughly 480,000 tokens) is the entire documentation corpus concatenated into one Markdown file, useful when a tool wants everything in a single fetch and has a context window large enough to hold it, but far too large to read speculatively the way the curated file is designed to be. Publishing both lets an agent choose the right tool for its own constraints instead of forcing every consumer through the same trade-off.

Criticism and Open Questions

The honest caveat, reported consistently in 2026 coverage, is that publishing an llms.txt and having it fetched are different things: major model providers’ own crawlers, including OpenAI’s, Google’s, and Anthropic’s, do not request /llms.txt in meaningful volume as of 2026. Its clearest proven users are IDE coding agents and documentation-aware assistants pointed explicitly at a doc site, not general web-crawling LLM training pipelines or live chat-assistant browsing. There’s also no verification mechanism: unlike robots.txt, which crawlers are expected to respect as a directive, llms.txt is purely descriptive and self-reported, so nothing stops a site from publishing a curated file that oversells its own content, and no authority checks accuracy. For teams weighing whether to invest in one, it’s best understood as a low-cost, plausible-upside bet for agent- and IDE-facing discoverability, not yet a guaranteed lever for general AI search visibility, which is a distinct problem covered under Generative Engine Optimization (GEO).

# Acme SDK

> Acme SDK is a typed client for the Acme API. This file
> is a machine-readable map of our docs for LLMs and
> coding agents; start here before crawling the full site.

Acme SDK supports Python, TypeScript, and Go. Auth uses
short-lived API keys, not OAuth.

## Docs

- [Quickstart](https://docs.acme.dev/quickstart.md): install, auth, first request
- [API Reference](https://docs.acme.dev/reference.md): every endpoint, typed
- [Rate limits](https://docs.acme.dev/limits.md): per-key quotas and backoff rules

## Examples

- [Python examples](https://docs.acme.dev/examples/python.md)
- [TypeScript examples](https://docs.acme.dev/examples/ts.md)

## Optional

- [Changelog](https://docs.acme.dev/changelog.md): skip unless asked about version history
- [Brand guidelines](https://docs.acme.dev/brand.md)