# AI crawler rules in the robots.txt of the Tranco top 20,000

Dataset behind the Fentner Group report
[57% of sites that block GPTBot leave ChatGPT-User unrestricted](https://fentner.com/reports/gptbot-block-leaves-chatgpt-open/)
(FG-26-09, 2 September 2026).

Measured on 2026-09-02. Sample: Tranco daily list `K9QPW` (generated 2026-09-01,
MD5 `ed970e1b159546a9ade66e72ae76eb3b`), ranks 1 to 20,000.

## Files

| file | what it is |
| --- | --- |
| `robots_20000.csv` | one row per domain: fetch outcome and the state of every AI crawler token |
| `lookalike_sites.csv` | every site whose robots.txt contains a product token with a non-ASCII character |
| `numbers.json` | every figure quoted in the report, with the counts behind it |
| `vendors.json` | the crawler token list of each vendor, with the documentation URL it came from |
| `METHOD.md` | what was run: sample, fetch settings, parser, definitions, exclusions |
| `robots_rfc9309.py` | the RFC 9309 parser |
| `analyse.py` | the analysis; reads the raw fetch, writes `numbers.json` and both CSVs |
| `fetch_dual.py` | the first fetcher (ranks 1–2,298) |
| `fetch_curl.py` | the second fetcher (the remainder) |
| `verify_examples.py` | re-fetches every site named in the report and saves the bytes |
| `evidence/` | the saved robots.txt of every site named in the report |

## robots_20000.csv columns

| column | meaning |
| --- | --- |
| `rank` | Tranco rank |
| `domain` | pay-level domain |
| `band` | rank band: `1-1,000`, `1,001-5,000`, `5,001-10,000`, `10,001-20,000` |
| `status_research` | HTTP status returned to the research user-agent (`-1` = no response) |
| `status_browser` | HTTP status returned to the browser user-agent |
| `served_to` | which response was parsed: `research`, `browser`, or `none` |
| `ua_discriminated` | 1 if the research agent was refused and the browser agent was served |
| `html_at_robots` | 1 if the body served at `/robots.txt` is an HTML page |
| `usable_robots` | 1 if a non-empty, non-HTML robots.txt was obtained |
| `n_groups` | number of rule groups parsed |
| `n_tokens` | number of distinct product tokens named |
| `star` | state of the `*` group |
| `ua:<token>` | state of that crawler token (one column per token in `vendors.json`) |
| `nonascii_tokens` | number of product tokens containing a non-ASCII character |

State values for `star` and every `ua:` column:

* `absent` — no group in the file names this token
* `allow` — a group names it and disallows nothing
* `partial` — a group names it and disallows some path, but not the site root
* `block` — a group names it and the site root is disallowed

## How the data was collected

Each domain was requested once at `https://<domain>/robots.txt` with an honest
research user-agent naming the study, and once with a browser user-agent, at 20
concurrent requests with an 8-second connect timeout and a 20-second total
timeout. The second fetch separates a site that has no robots.txt from a site
that has one but will not serve it to an unrecognised agent. Files were parsed
per RFC 9309: case-insensitive product tokens, group semantics, merged groups,
`*` fallback, longest-match rule evaluation with `allow` winning ties.

Full detail, including the exact definitions used for "blocks" and "restricted",
is in `METHOD.md`.

## What is not here

The raw response bodies are not published. The corpus is roughly 500 MB and much
of it is other people's copyrighted boilerplate; the parsed state of every
crawler token is published instead, which is what every figure in the report is
computed from. The robots.txt of each site named in the report is published in
full under `evidence/`.

No personal data, credentials or API keys appear in any file.

## Licence

CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Attribute to Fentner
Group and link to the report.