A Standard Almost Nobody Reads

A proposed standard can be adopted widely and consumed by nothing. llms.txt is the clean instance. Ahrefs, measuring server logs across its own analytics customers, found that 28% of the 137,210 domains using Ahrefs Web Analytics publish an llms.txt file during May 2026, and that 97% of those files received zero traffic in the month. Publication and consumption are separate measurements, and here they point in opposite directions.

What the file was for

The proposal is recent and its origin is documented. Jeremy Howard of Answer.AI wrote in September 2024 that those interested in providing LLM-friendly content should add a /llms.txt file to their site, and the specification site repeats the framing: a markdown file at /llms.txt to provide LLM-friendly content.

Its distinguishing property is curation. robots.txt governs access and states what a crawler may fetch. sitemap.xml is exhaustive and states what exists. llms.txt is neither. The spec's argument is that site authors know best, and can provide a list of content that an LLM should use — an editorial selection, published by the site, of what it would prefer a machine to read. That is a different kind of file from the two it sits beside, and its value depends entirely on something choosing to fetch it.

Adoption

Two independent counts agree that publication is real and rising. Ahrefs' figure of 28% is the higher of the two, and it is an upper bound rather than a web-wide rate: the sample is sites running Ahrefs Web Analytics, which the publisher states skews technical. It is not 28% of the web.

Originality.AI, tracking a stated sample of more than three million websites, counted 36,120 sites with an llms.txt in May 2026 against 4,088 a year earlier, an 8.8-fold increase, rising to 38,980 when llms-full.txt and ai.txt are included. The page gives limited methodology, so the counts belong to Originality.AI by name. The two measurements are not comparable to each other — one is a share of a technical customer base, the other a raw count across a larger sample — but both describe growth.

Vendor behaviour matches. Anthropic's developer documentation serves an llms.txt at platform.claude.com/llms.txt, 73,393 bytes, beginning "# Anthropic Developer Documentation"; Cloudflare's serves one at developers.cloudflare.com/llms.txt. Both were confirmed live on 2 September 2026. Publishing a file is evidence of publication and nothing more.

Consumption

The traffic measurement is the part that settles it. Ahrefs reports that 96% of requests to llms.txt files came from bots rather than people, and that 77% of the bots reading llms.txt are not from AI tools at all. Among the files that received any request in May 2026, all AI bot categories together accounted for 19.5% of requests, and the AI retrieval agents specifically — OAI-SearchBot, PerplexityBot and Claude's search crawler — accounted for 1.1% of total requests.

The 97% figure needs reading carefully. It is not a claim that the files are unusable or malformed. It is a count of files that were never requested at all during the measured month, by anything, human or machine. A file with no requests has not been evaluated and rejected. It has not been looked for.

One further observation in the same study is more diagnostic than any percentage. Ahrefs found that zero requests came from AI bots for llms.txt files that do not exist. A crawler that expected the file would probe for it and record a 404. Nothing probed. The convention is not part of any retrieval routine that touched this sample.

What the platforms say, and do not say

Google is on record twice. In June 2025 John Mueller, Search Advocate at Google, wrote on Bluesky that no AI system currently uses llms.txt. Google's own documentation, last updated in July 2026, states that you do not need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search, and adds that doing so will neither harm nor help visibility because Google Search ignores them.

No equivalent statement from OpenAI, Anthropic or Microsoft could be located. That is an absence of evidence, not a denial. Those firms have said nothing publicly about whether they consume the file, and the honest position is that the question is unanswered for every operator except Google. The log measurement above is what stands in for a statement: whatever the intent, the requests are not arriving.

Implication

The case is worth keeping because it is closed and cheap. A file was proposed, documented, adopted by tens of thousands of sites including the documentation properties of the model vendors themselves, and read by almost nothing. The cost of publishing it was small and so is the loss. The instructive part is the reasoning that produced the effort: the file was adopted because it was plausible that machines would want it, not because any measurement showed that machines fetched it.

The distinction the case turns on is between a signal an organisation emits and a signal something consumes. Only the second has any effect, and only the second is hard to arrange. Emission is under the publisher's control and costs an afternoon. Consumption depends on a retrieval routine written by someone else, and no amount of care in the emitting changes it.

The same test applies to every machine-facing artefact an organisation maintains. Two conditions separate the ones that work from the ones that do not: a named consumer states on the record that it reads the artefact, and the server logs show that consumer fetching it. Both are checkable from material an organisation already holds. Access logs are the cheapest instrument available for the purpose and are rarely inspected for it.

Nothing here argues that llms.txt will remain unread. Adoption is growing and consumption could follow at any point, which is precisely why the measurement should be repeated rather than assumed. What is not supportable is treating publication as evidence of uptake. Those are separate facts, they are separately countable, and at present they disagree.