# Method

What was run, in the order it was run, so the measurement can be repeated.

## 1. Sample

Tranco daily list, generated 2026-09-01, list ID `K9QPW`
(https://tranco-list.eu/list/K9QPW). Downloaded from
`https://tranco-list.eu/download/K9QPW/1000000`; the file is 22,707,157 bytes,
MD5 `ed970e1b159546a9ade66e72ae76eb3b`. Tranco combines Cloudflare Radar, Chrome
UX Report, Farsight, Majestic and Umbrella rankings over a 30-day window using
the Dowdall rule, with pay-level-domain filtering on.

The study uses ranks 1 to 20,000. Rank bands used throughout: 1–1,000,
1,001–5,000, 5,001–10,000, 10,001–20,000.

## 2. Fetch

Date of measurement: 2026-09-02, between 13:53 and 15:40 CEST, with the named-site evidence captured at the end of that window.

Each domain was requested at `https://<domain>/robots.txt`. If the connection
failed outright, `https://www.<domain>/robots.txt` was tried once. Every domain
was then fetched a second time at the same URL with a browser user-agent.

* Research agent:
  `FentnerResearchBot/1.0 (+https://fentner.com/research; robots.txt measurement study; contact: research@fentner.com)`
* Browser agent:
  `Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36`

Settings: 20 concurrent requests across the whole run, one domain fetched once
per agent, 8-second connect timeout, 20-second total timeout per request,
redirects followed to a depth of 5, response truncated at 300,000 characters.

The second fetch exists to separate two cases that look identical from one
agent: a site that has no robots.txt, and a site that has one but refuses to
serve it to a crawler it does not recognise. Where the research agent was
refused and the browser agent was served, the browser response is used and the
domain is flagged `ua_discriminated` in the dataset. Where neither agent was
served a 200, the domain is excluded from every rate in the report rather than
counted as a site with no AI rules.

Fetches that failed direct were repeated through a rotating HTTP proxy
(`proxy_refetch.py`), which assigns a fresh IP per request. A 403 or a timeout
from one address is a fact about that address, not about the site, so every
domain that returned nothing usable to either agent, or that returned a refusal
or rate-limit status (401, 403, 406, 408, 409, 425, 429 or 5xx) to either agent,
was fetched again from a fresh IP with both user-agents, retrying once on a
status a different IP might change. 5,921 domains qualified. The proxy result
replaces the direct one only when it produced a 200, which happened for 729
domains; 291 of those had returned nothing usable to either agent directly and
are the domains the proxy recovered, and the `fetched_via` column records which
result was used. The user-agent-discrimination
control was re-run the same way, so the 389 domains recorded as refusing the
research agent refused it from several addresses. The proxy URL is read from the
environment variable `PROXY_URL` and appears in no script, output or dataset
file. The sites named in the report were re-fetched through the same proxy.

The first 2,298 domains were fetched with a Python `urllib` client
(`fetch_dual.py`); that client stalled on a server that trickled bytes below its
per-read timeout, so the remaining 17,702 were fetched with the same request
settings through `curl`, which enforces a hard wall-clock limit
(`fetch_curl.py`). Both raw files are analysed together and have identical
fields.

## 3. Parsing

`robots_rfc9309.py` implements the parts of RFC 9309 the study depends on:

* Field names are matched case-insensitively; product tokens are compared
  case-insensitively (section 2.1).
* A group is one or more consecutive `user-agent` lines followed by that group's
  rules. A `user-agent` line appearing after a rule line starts a new group
  (section 2.2.1).
* Where several groups name the same product token, their rules are merged
  (section 2.2.1).
* A crawler that is not named by any group falls back to the `*` group
  (section 2.2.1).
* Path matching supports `*` and `$`. The longest matching rule wins, and
  `allow` wins a tie (section 2.2.2).

Everything after `#` on a line is discarded. A leading byte-order mark is
stripped. Lines without a `field: value` shape are counted but do not affect
rule evaluation.

## 4. Definitions

For a domain and a crawler token:

| term | meaning |
| --- | --- |
| `absent` | no group in the file names this token |
| `allow` | a group names it and disallows nothing |
| `partial` | a group names it and disallows some path, but not the site root |
| `block` | a group names it and the site root is disallowed |

Two derived predicates are used, and the difference between them matters:

* **blocks** — the file *names* the crawler and disallows it something
  (`block` or `partial`). This is the deliberate act being counted: somebody
  wrote a rule about this bot.
* **restricted** — the crawler is in fact kept out of something: either the file
  names it and disallows it, or the file never names it and the `*` group
  disallows the whole site. A site with a blanket `Disallow: /` for `*` restricts
  every crawler, named or not, and is not counted as leaving anything open.

A site is counted as having a **usable robots.txt** if either agent received
HTTP 200 with a non-empty body that is not HTML.

## 5. Crawler token lists

Every token list was read from the vendor's own current documentation on
2026-09-02, not from a third-party bot directory. The doc URL for each vendor is
carried in `vendors.json` and cited in the report. Tokens are classified as
`training` (collected in advance to train models), `search` (collected in
advance to build an index), `user` (fetched while answering a person's question)
or `other`.

The gap measured in the report pairs, per vendor, the bulk crawler against that
vendor's live siblings:

| vendor | bulk | live siblings | of which user-triggered |
| --- | --- | --- | --- |
| OpenAI | GPTBot | OAI-SearchBot, ChatGPT-User | ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot, Claude-User | Claude-User |
| Meta | meta-externalagent | meta-webindexer, meta-externalfetcher | meta-externalfetcher |
| Amazon | Amazonbot | Amzn-SearchBot, Amzn-User | Amzn-User |
| Mistral | MistralAI-Training | MistralAI-Index, MistralAI-User | MistralAI-User |
| Perplexity | PerplexityBot | Perplexity-User | Perplexity-User |
| Google | Google-Extended | Googlebot | — |
| Apple | Applebot-Extended | Applebot | — |

The first five are the vendors used in the headline figure, because each runs a
bulk crawler and a documented user-triggered fetcher under separate tokens.
Perplexity is reported separately because it documents no training crawler at
all. Google and Apple are reported separately because their "extended" tokens
are opt-out controls rather than crawlers, and their live sibling is the ordinary
search crawler.

`anthropic-ai` and `Claude-Web` are tracked as `undocumented`: they appear in
robots.txt files in the wild but are not named anywhere in Anthropic's current
crawler documentation. Anthropic publishes no statement retiring them, so the
report says they are undocumented, not that they are retired.

## 6. Named examples

Every site named in the report was re-fetched at write time by
`verify_examples.py`. Each fetch is saved under `evidence/<domain>.txt` with the
timestamp, final URL, HTTP status, user-agent used, byte count and SHA-256 of
the body, followed by the body. The claim printed for each site is recomputed
from the file that was just saved. Any site that could not be re-fetched was
dropped from the report.

## 7. Exclusions

* Domains where neither agent returned HTTP 200 are excluded from all rates.
* Domains serving HTML at `/robots.txt` are excluded from all rate denominators
  and counted separately, because the operator has no robots.txt at all whatever
  the status code says.
* Empty bodies are treated as no file.
* No domain is excluded for its content, its rank, or the rules it contains.

## 8. What was not measured

No crawler was observed. This is a measurement of what site operators wrote,
checked against what the vendors document. It does not test whether any bot
obeys its own rules, and no page was requested from any AI product to see
whether it could read a blocked site.

## 9. Files

| file | what |
| --- | --- |
| `fetch_dual.py` | first fetcher (ranks 1–2,298) |
| `fetch_curl.py` | second fetcher (the remainder), hard per-request timeout |
| `proxy_refetch.py` | re-fetches everything the direct run could not settle, via rotating proxy |
| `render_report.py` | fills the report text from `numbers.json` so no figure is typed by hand |
| `robots_rfc9309.py` | the parser |
| `vendors.json` | verified per-vendor token lists and doc URLs |
| `analyse.py` | all analysis; writes `numbers.json` |
| `verify_examples.py` | re-fetches every named site and saves the evidence |
| `ngram_check.py` | 5-gram overlap check against the other reports |
| `robots_20000.csv` | one row per domain, per-crawler state |
| `lookalike_sites.csv` | every site with a non-ASCII character in a product token |
| `numbers.json` | every figure quoted in the report |
| `evidence/` | the saved robots.txt of every named site |