What is llms.txt, and does it work?

llms.txt is a proposed markdown file, published at your domain root, that hands AI systems a curated map of your most important pages. Jeremy Howard introduced it in September 2024. As of 2026 it is widely published but rarely read. The server logs are blunt: the major AI crawlers almost never request the file, and no big AI vendor has said it uses one.

What is llms.txt?

It is a plain-text file written in markdown, placed at /llms.txt, that lists a site's key pages with short descriptions so a language model can find them without crawling the whole domain. The spec asks for one H1 with the project name, a blockquote summary, then H2 sections of annotated links. Answer.AI proposed it in 2024.

The idea came from Jeremy Howard, known for fast.ai and Answer.AI, in a post dated September 3, 2024. The published spec states the problem plainly: "constructing the right context for LLMs based on a website is ambiguous," and context windows are "too small to handle most websites in their entirety." A short, human-curated index is meant to fix that.

The format is strict about order. Only the H1 is required; everything else is optional but expected: a blockquote summary, free-form markdown sections, then H2 headers each containing a list of [name](url) links with optional notes. The spec also describes a fuller variant that inlines the actual page text instead of only linking to it. In practice most sites publish that second file as llms-full.txt.

One line in the spec matters more than any format detail. The authors write that they expect llms.txt "will mainly be useful for inference, i.e. at the time a user is seeking assistance, as opposed to for training." It was never pitched as a training-crawler control. It was pitched as a convenience for a model already trying to answer a question about you.

How is llms.txt different from robots.txt?

They solve opposite problems. robots.txt tells crawlers what they are allowed to touch, and well-behaved bots check it by default. llms.txt suggests what is worth reading, and it is read on demand, if at all. One is an access rule from 1994 with a formal standard behind it. The other is a two-year-old content hint with no vendor commitment.

The spec itself draws the line: robots.txt "is generally used to let automated tools know what access to a site is considered acceptable, such as for search indexing bots," while llms.txt "information will often be used on demand when a user explicitly requests information about a topic." One governs permission; the other offers curation.

If you want the compliance picture for the older file, we cover it in Do AI crawlers respect robots.txt?. The short version is that even the enforceable-looking file is only a request. llms.txt is one step softer: a suggestion nobody agreed to read.

Do AI crawlers actually read llms.txt?

The measured answer is mostly no. Ahrefs looked at 137,210 domains and found that 97% of published llms.txt files got zero requests in May 2026. OtterlyAI logged 62,100 AI-bot visits over 90 days and saw only 84 of them touch the file, about 0.1%. The bots that do exist are fetching your HTML, not your index.

The Ahrefs study, published in June 2026, is the largest of its kind. Of the files that did receive any traffic, most requests came from SEO audit tools rather than AI systems; the retrieval bots that actually power ChatGPT and Perplexity answers made up a sliver of the hits. A separate 90-day test by OtterlyAI found no correlation at all between having the file and any change in AI crawler activity.

The citation question has been tested too, not just the traffic one. SE Ranking analyzed roughly 300,000 domains and concluded the file "doesn't impact how AI systems see or cite your content today." Publishing one did not move how often a site got cited in AI answers. That is the outcome most people who add the file are actually hoping for, and the data does not support it yet.

Has any AI company said it uses llms.txt?

No major vendor has committed to reading it. Google has been the most direct: John Mueller compared llms.txt to the long-dead keywords meta tag, and Gary Illyes confirmed Google Search does not support it. OpenAI, Anthropic, and Perplexity publish crawler guidance built around robots.txt, not llms.txt, and none has stated its models read llms.txt from other sites.

Mueller's comment is worth reading in full, because it names the same server-log gap the studies later measured: "none of the AI services have said they're using LLMs.TXT (and you can tell when you look at your server logs that they don't even check for it)." His comparison was blunt: "it's comparable to the keywords meta tag," a signal search engines abandoned because it was self-declared and easy to game.

There is one honest wrinkle. Google Search dismisses the file, but Google Chrome does not: Lighthouse added an Agentic Browsing audit that checks whether a site publishes an llms.txt to help agents understand its structure. So the same company treats the file as noise for ranking and as a nice-to-have for tool-driven browsing. Both positions can be right, because they describe different jobs.

If almost nobody reads it, why is adoption climbing?

Because it is cheap to add, easy to auto-generate, and low-risk to be early on. Adoption of a valid llms.txt across the top 10,000 sites rose from 1.04% to 5.61% between mid-2025 and mid-2026, and doc platforms now ship the file by default. Growth reflects supply and optimism, not proof that the audience it targets is reading it.

The HTTP Archive analysis by Casey Burridge put the top-10,000 figure at 5.61% by June 2026, up more than fivefold in a year. Originality.ai, tracking more than three million sites, reported the raw count of llms.txt files growing 8.8 times over the same window. A large share of that comes from platforms, not per-site decisions: Mintlify auto-generates an llms.txt for every documentation site it hosts, so thousands of sites acquired the file without anyone choosing it.

That is a familiar pattern. A low-cost convention spreads fast because being early costs almost nothing, well ahead of any evidence that the intended readers have shown up. Rising adoption and near-zero reads are not a contradiction; they are what a hopeful standard looks like in its first two years.

Where does llms.txt actually help today?

In developer tooling, where a person points an agent at a specific docs site. Coding assistants and agentic browsers can fetch llms.txt on demand as a navigation map, which is the inference-time use the spec described. That is why Anthropic, Stripe, and Vercel maintain the file for their own developer docs, even as the broader "get cited in AI answers" pitch stays unproven.

This narrow case is where the evidence turns mildly positive. When you tell a coding tool to "use the Stripe docs," an on-demand fetch of a curated index is useful, and it matches the on-demand framing in the original proposal. It is a different job from an autonomous search crawler indexing the open web at scale, which is the job the citation studies measured and found no lift for. Anthropic, for one, publishes its own file at platform.claude.com/docs/llms.txt, which tells you the file is useful for reaching a model with its docs, not that the model reads yours.

Should you publish an llms.txt file?

If your platform generates one for free, keep it; the cost is zero and the developer-tooling upside is real. Do not expect it to earn AI citations on its own, because no current data shows that it does. The things that get you cited are crawlable server-rendered HTML, self-contained answer passages, and third-party corroboration, none of which an index file replaces.

The order of operations is what matters. An llms.txt cannot help a model that cannot fetch and parse your actual pages in the first place, so the fundamentals come before the index: let the AI crawlers in, serve content server-side, and write passages a model can lift. Those are the moves that move the number, and we rank them in How to get cited by ChatGPT and frame the whole discipline in What is answer engine optimization?. An llms.txt is a reasonable finishing touch, not a foundation.

The honest close is a measurement one. If you add an llms.txt hoping for more AI citations, the only way to know whether it did anything is a dated, repeatable before-and-after, because a single AI answer proves nothing on its own. A free BeCited report scores whether AI crawlers can actually reach and parse your pages, which is the thing an index file cannot fix, and the methodology page lists exactly what is measured live. Nothing unmeasured is presented as proof, and for llms.txt specifically, the proof is not in yet.