What it is
llms.txt is a plain-text file, written in Markdown, that a website can publish at the address yoursite.com/llms.txt. It is meant to give AI systems — chatbots, coding assistants, autonomous agents — a short, curated map of a site’s most useful content: an H1 title, a one-line summary, and a set of linked sections pointing to documentation, policies, or other material the site’s owners consider important for an AI system to read. The idea, as the specification puts it, is that “web pages are built for people” — full of navigation menus, advertising, and scripts — and that converting that clutter back into clean text for a language model is “difficult and imprecise.” llms.txt is a shortcut around that problem: a hand-curated, AI-facing summary that sits alongside the page built for humans.
The comparison most often drawn is to robots.txt, the decades-old convention that tells search-engine crawlers which parts of a site they may index. But the specification itself draws a sharper line: robots.txt is about permission to crawl, and XML sitemaps are exhaustive lists of every page; llms.txt is neither. It is meant to be consulted selectively, “on demand, when an agent needs information about a topic while assisting a user,” rather than crawled wholesale.
The format was proposed by Jeremy Howard, the Australian machine-learning researcher and co-founder of Answer.AI (also known for fast.ai), who published the original specification on September 3, 2024, at llmstxt.org. A revised version 2, reflecting roughly two years of real-world use, followed in 2026.
Why it matters for AI governance and narratives
llms.txt sits at the center of a live framing contest about who gets to broker the relationship between the web and the machines reading it. Proponents — documentation platforms, AI labs’ own developer-relations teams, and a fast-growing tier of “AI SEO” consultants — present it as the natural next step after robots.txt: a small, voluntary courtesy that lets any site opt into the machine-readable web on its own terms. That framing implicitly treats the arrival of AI crawlers as settled fact, and positions the file as evidence of cooperative, well-governed adaptation rather than something being done to publishers without consent or compensation.
The counter-evidence is pointed and comes from the same side of the industry doing the measuring. An Ahrefs analysis of roughly 137,000 domains found that 97% of published llms.txt files received zero requests in a sampled month — no bot traffic, no human traffic. Google’s John Mueller has publicly likened the file to the long-dead “keywords” meta tag, noting that server logs show major AI services do not appear to check for it at all. No major AI lab — OpenAI, Anthropic, or Google — has officially confirmed its crawlers consult the file, even though all three publish their own llms.txt for their developer documentation. That gap — widespread voluntary publication, contested or absent actual use — is itself a narrative signal: it suggests adoption driven by anticipatory hope and SEO-adjacent incentive rather than by any demonstrated technical necessity, at least so far.
Key facts and dates
- Origin: Proposed by Jeremy Howard (Answer.AI) on September 3, 2024; specification maintained at llmstxt.org, with a v2 update in 2026.
- Format: Markdown file at /llms.txt — H1 title, blockquote summary, optional detail, H2-delimited link sections. A companion llms-full.txt sometimes carries the complete expanded content.
- Adoption: Thousands of sites, and documentation platforms such as Mintlify generate the file automatically; Anthropic has partnered with Mintlify on this for its own docs. Independent estimates of valid-file prevalence vary widely by sample — one large-scale study put it near 28% of analyzed domains, skewed toward AI-aware, tech-savvy publishers.
- Disputed utility: No major AI lab has officially confirmed its models or agents read the file as part of normal operation; third-party server-log and traffic studies offer conflicting readings, from near-total neglect (Ahrefs) to meaningful agent visits (Profound, a marketing-analytics firm with a commercial stake in the claim).
The observatory’s own wire has flagged llms.txt as one marker among several — alongside agent-rating services and infrastructure rebuilds to handle machine-generated commits — in an assembling “machine-to-machine” layer of the web. Whether llms.txt is infrastructure for that layer or a protocol in search of a user remains genuinely unresolved.
Where to learn more
- The /llms.txt file, v2 — llmstxt.org (primary specification)
- What Is llms.txt, and Should You Care About It? — Ahrefs (large-scale adoption and usage study, skeptical findings)
- What is llms.txt? Breaking down the skepticism — Mintlify (adoption from a documentation-platform perspective)
- Meet llms.txt, a proposed standard for AI website content crawling — Search Engine Land (trade-press overview)