Who Is Shut Out by robots.txt

Access to an organisation's published material by AI systems is governed by a text file most sites never edit. Cloudflare retrieved robots.txt for the top 10,000 domains by its own Radar ranking in mid-2025, found a file on 3,816 of them, and reported that 546, about 14%, contained an allow or disallow directive for an AI bot. Six in seven of the largest sites with a robots.txt had not addressed the question at all. The minority that had addressed it did so decisively, and unevenly.

Blocking is concentrated, not general

Among those 546 domains, the same Cloudflare analysis from July 2025 found that GPTBot was the most frequently blocked agent, disallowed by 312 domains, 250 fully and 62 partially, with CCBot and Google-Extended next. GPTBot was also the most explicitly permitted agent, allowed by 61 domains, 18 fully and 43 partially. The same name leads both lists. Large publishers are not converging on a policy; they are making opposite decisions about the same crawler.

Away from the largest sites the picture inverts. Ahrefs examined approximately 140 million websites in May 2025 and found that GPTBot is the most blocked AI bot, with 5.89% of all websites blocking it, rising to 7.3% when measured across all 461 million robots.txt files including subdomains. Ahrefs counts explicit robots.txt directives only, not firewall or IP-level blocking, so the figure is a floor for restriction overall and an accurate count of the file-based sort. The two measurements are consistent and the gap between them is the finding: on the largest sites blocking is a live editorial decision, and across the web it is a rounding error.

Direction of travel is measurable too. Ahrefs reports that ClaudeBot saw the highest growth in block rates, increasing by 32.67% over the year to May 2025. Both figures are Ahrefs' own measurement, published by a vendor that sells crawl and index data.

What is actually crawling

Restriction should be read against volume. Cloudflare's May 2025 crawler traffic share puts Googlebot at 50%, Bingbot at 8.7%, GPTBot at 7.7% and ClaudeBot at 5.4%. Growth from May 2024 to May 2025 was 305% for GPTBot against 96% for Googlebot, with total AI and search crawler traffic up 18% over full-month periods. Cloudflare sampled fixed customer cohorts to remove the effect of its own customer growth.

Two things follow. Search crawling still dominates the volume, so a directive aimed at AI agents governs a minority of requests. And the agents growing fastest are the ones most often named in the blocks, which means the population subject to restriction and the population expanding are the same population.

Precise naming matters when writing these directives. The agents documented across these sources are GPTBot, ChatGPT-User, OAI-SearchBot, CCBot, ClaudeBot, anthropic-ai, Google-Extended, PerplexityBot, Bytespider, Applebot-Extended, Amazonbot and Meta-ExternalAgent. They are not interchangeable. Several operators run separate agents for training and for answering a live query, and a rule written against one leaves the other untouched.

The restriction is not evenly distributed

The most consequential measurement is not a percentage of sites. The Data Provenance Initiative, in a longitudinal audit of 14,000 web domains underlying the C4, RefinedWeb and Dolma corpora, found that within a single year robots.txt changes had rendered 5% or more of all tokens in C4, and 28% or more of its most actively maintained critical sources, fully restricted from use. The study, published in July 2024 with 49 authors, also reports that 45% of C4 is restricted by terms of service.

The ratio between those two figures is the point. On the study's own numbers, restriction runs roughly five times heavier on the sources a system would most want than across the corpus at large. The web is not closing uniformly; it is closing at the top. A crawler operating under these constraints reaches a corpus whose best-maintained material is disproportionately missing, and its answers are assembled from what remains.

The default has also moved. Cloudflare announced on 1 July 2025 that it was changing the default to block AI crawlers unless they pay creators for their content. The post states no figure for how many domains that applies to and none should be supplied. What can be said is that the decision moved from the site operator to the infrastructure provider for an unstated share of the web.

Implication

For most organisations the operative fact is inaction. A file exists, it was generated by a platform or copied from a template, and no one has read it since. On the Cloudflare measurement that describes the overwhelming majority even of the largest sites. Whatever access policy an organisation believes it has, the default is almost certainly what it actually has, and the first useful step is to read the file and know which of the agents above are currently permitted.

The second point is that the choice is not binary in the way the debate presents it. The agents split by purpose. An organisation can decline training use and permit retrieval for live answers, or the reverse, because the operators publish separate user agents for the two. That distinction is available in the file and is missed by anyone writing a single blanket rule.

Third, robots.txt is a request, not a control. It is honoured by declared, well-behaved agents and is silent about anything that crawls without identifying itself. Ahrefs measures directives; it does not measure compliance, and no source here does. An organisation that needs enforcement rather than a preference is looking at infrastructure, not at a text file — which is the change Cloudflare's default represents.