| File | Contents |
|---|---|
README.md | Column definitions, collection method, licence |
METHOD.md | Sample, fetch settings, parser, definitions, exclusions |
robots_20000.csv | One row per domain: fetch outcome and per-crawler state |
lookalike_sites.csv | Product tokens containing non-ASCII characters |
numbers.json | Every figure quoted in the report |
vendors.json | Per-vendor crawler tokens and the documentation URL each came from |
robots_rfc9309.py | RFC 9309 parser |
analyse.py | Analysis; writes numbers.json and both CSVs |
fetch_dual.py | First fetcher (ranks 1–2,298) |
fetch_curl.py | Second fetcher (the remainder) |
proxy_refetch.py | Re-fetch of everything the direct run could not settle |
verify_examples.py | Re-fetch and evidence capture for every named site |
render_report.py | Fills the report text from numbers.json |
ngram_check.py | 5-gram overlap check |
evidence/ | Saved robots.txt of all 37 sites named in the report |