{
 "openai": {
  "docs": [
   "https://developers.openai.com/api/docs/bots"
  ],
  "notes": "The doc explicitly separates user-triggered fetching from crawling: ChatGPT-User is not governed by GPTBot's rules, and OpenAI warns robots.txt may not apply at all to it. Full UA string documented is ChatGPT-User/1.0. As of 2026-09-02 the page does NOT document 'ChatGPT-User/2.0', 'ChatGPT Agent', or 'Operator' as separate robots.txt tokens - agent/browsing traffic is still documented under ChatGPT-User. OAI-AdsBot is a token often missing from third-party lists.",
  "quotes": [
   {
    "text": "Because these actions are initiated by a user, robots.txt rules may not apply.",
    "url": "https://developers.openai.com/api/docs/bots"
   }
  ],
  "tokens": [
   {
    "token": "GPTBot",
    "kind": "training",
    "role": "Crawls content used to train OpenAI's generative AI foundation models."
   },
   {
    "token": "OAI-SearchBot",
    "kind": "search",
    "role": "Crawls to surface and link websites in ChatGPT search results; not for training."
   },
   {
    "token": "ChatGPT-User",
    "kind": "user",
    "role": "Fetches a page when a ChatGPT user or a Custom GPT action triggers a live visit."
   },
   {
    "token": "OAI-AdsBot",
    "kind": "other",
    "role": "Validates the safety of web pages submitted as ads in ChatGPT."
   }
  ],
  "gap": {
   "bulk": [
    "GPTBot"
   ],
   "live": [
    "OAI-SearchBot",
    "ChatGPT-User"
   ],
   "user": [
    "ChatGPT-User"
   ],
   "headline": true
  }
 },
 "anthropic": {
  "docs": [
   "https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler"
  ],
  "notes": "Anthropic is the outlier: it does NOT carve out user-initiated fetches from robots.txt. The page states its bots honor industry-standard robots.txt directives and gives a separate User-agent: Claude-User / Disallow: / block recipe, i.e. Claude-User is a distinct agent that obeys its own rules rather than ClaudeBot's. Blocking ClaudeBot does not block Claude-User or Claude-SearchBot. The current article does NOT mention 'anthropic-ai' or 'Claude-Web'; they were the pre-ClaudeBot identifiers and are simply absent from (dropped out of) the live doc - I could not find an Anthropic page that explicitly declares them retired, only third-party reporting. Treat 'anthropic-ai'/'Claude-Web' as undocumented legacy, not as vendor-confirmed retired. 'claude-code' also appears in third-party lists but is NOT in this support article.",
  "quotes": [
   {
    "text": "Anthropic's Bots respect \"do not crawl\" signals by honoring industry standard directives in robots.txt.",
    "url": "https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler"
   },
   {
    "text": "Claude-User allows site owners to control which sites can be accessed through these user-initiated requests.",
    "url": "https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler"
   }
  ],
  "tokens": [
   {
    "token": "ClaudeBot",
    "kind": "training",
    "role": "Collects web content to train/improve Anthropic's generative AI models."
   },
   {
    "token": "Claude-SearchBot",
    "kind": "search",
    "role": "Crawls the web to improve search result quality inside Claude."
   },
   {
    "token": "Claude-User",
    "kind": "user",
    "role": "Fetches a page in real time when a Claude user's question requires it."
   },
   {
    "token": "anthropic-ai",
    "kind": "undocumented",
    "role": "not named in Anthropic's current documentation"
   },
   {
    "token": "Claude-Web",
    "kind": "undocumented",
    "role": "not named in Anthropic's current documentation"
   }
  ],
  "gap": {
   "bulk": [
    "ClaudeBot"
   ],
   "live": [
    "Claude-SearchBot",
    "Claude-User"
   ],
   "user": [
    "Claude-User"
   ],
   "headline": true
  }
 },
 "perplexity": {
  "docs": [
   "https://docs.perplexity.ai/guides/bots"
  ],
  "notes": "Strongest carve-out of any vendor: Perplexity-User is documented as generally ignoring robots.txt entirely, so neither PerplexityBot's rules nor a Perplexity-User rule are guaranteed to be honored. Perplexity documents no separate training crawler at all.",
  "quotes": [
   {
    "text": "Since a user requested the fetch, this fetcher generally ignores robots.txt rules.",
    "url": "https://docs.perplexity.ai/guides/bots"
   }
  ],
  "tokens": [
   {
    "token": "PerplexityBot",
    "kind": "search",
    "role": "Crawls to surface and link sites in Perplexity search results; explicitly not for foundation-model training."
   },
   {
    "token": "Perplexity-User",
    "kind": "user",
    "role": "Visits a page on demand when a user's question triggers a live fetch."
   }
  ],
  "gap": {
   "bulk": [
    "PerplexityBot"
   ],
   "live": [
    "Perplexity-User"
   ],
   "user": [
    "Perplexity-User"
   ],
   "headline": false
  }
 },
 "google": {
  "docs": [
   "https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers",
   "https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers",
   "https://developers.google.com/search/docs/crawling-indexing/google-user-triggered-fetchers",
   "https://developers.google.com/search/docs/appearance/ai-features"
  ],
  "notes": "Google's three-way taxonomy is common crawlers / special-case crawlers / user-triggered fetchers, and the user-triggered class as a whole is documented as generally ignoring robots.txt. Google-Extended is a control token, not a crawler: it governs Gemini model training and grounding in Gemini Apps + Grounding with Google Search on Vertex AI. It does NOT control AI Overviews or AI Mode - those are served out of the Googlebot Search index, so the only lever is Googlebot's robots.txt rules plus snippet controls (nosnippet, data-nosnippet, max-snippet, noindex). Google-Extended also has zero effect on Search inclusion or ranking. Note 'Google-Agent' is newly present in the user-triggered fetchers list and is easy to miss.",
  "quotes": [
   {
    "text": "Because the fetch was requested by a user, these fetchers generally ignore robots.txt rules.",
    "url": "https://developers.google.com/search/docs/crawling-indexing/google-user-triggered-fetchers"
   },
   {
    "text": "Google-Extended does not impact a site's inclusion in Google Search nor ranking.",
    "url": "https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers"
   },
   {
    "text": "AI is built into Search and integral to how Search functions",
    "url": "https://developers.google.com/search/docs/appearance/ai-features"
   },
   {
    "text": "robots.txt directives for Googlebot is the control for site owners to manage access",
    "url": "https://developers.google.com/search/docs/appearance/ai-features"
   },
   {
    "text": "a page must be indexed and eligible to be shown in Google Search with a snippet",
    "url": "https://developers.google.com/search/docs/appearance/ai-features"
   }
  ],
  "tokens": [
   {
    "token": "Googlebot",
    "kind": "search",
    "role": "Main Search crawler; its index also feeds AI Overviews and AI Mode."
   },
   {
    "token": "Google-Extended",
    "kind": "training",
    "role": "Opt-out control for Gemini model training and Gemini/Vertex grounding; not a crawler itself."
   },
   {
    "token": "GoogleOther",
    "kind": "other",
    "role": "Generic fetcher used by various Google product teams for public content."
   },
   {
    "token": "Google-NotebookLM",
    "kind": "user",
    "role": "User-triggered fetch when a user adds a URL source to NotebookLM."
   },
   {
    "token": "Google-CloudVertexBot",
    "kind": "other",
    "role": "Site-owner-requested crawls for building Vertex AI Agents."
   },
   {
    "token": "Google-Agent",
    "kind": "user",
    "role": "User-triggered fetch performed by a Google agentic product on a user's behalf."
   }
  ],
  "gap": {
   "bulk": [
    "Google-Extended"
   ],
   "live": [
    "Googlebot"
   ],
   "user": [],
   "headline": false
  }
 },
 "apple": {
  "docs": [
   "https://support.apple.com/en-us/119829"
  ],
  "notes": "Applebot-Extended never fetches anything; it is purely a usage-permission signal read from robots.txt. Apple documents no separate user-triggered fetch token, so Apple makes no user-initiated robots.txt carve-out statement. Blocking Applebot removes you from Siri/Spotlight/Safari search too - it is not a training-only lever.",
  "quotes": [
   {
    "text": "publishers can choose to opt out of their website content being used to train",
    "url": "https://support.apple.com/en-us/119829"
   }
  ],
  "tokens": [
   {
    "token": "Applebot",
    "kind": "search",
    "role": "Real crawler for Siri/Spotlight/Safari search; crawled data may also feed Apple foundation-model training."
   },
   {
    "token": "Applebot-Extended",
    "kind": "training",
    "role": "Control-only token; does not crawl - opts already-crawled content out of Apple foundation-model training."
   }
  ],
  "gap": {
   "bulk": [
    "Applebot-Extended"
   ],
   "live": [
    "Applebot"
   ],
   "user": [],
   "headline": false
  }
 },
 "meta": {
  "docs": [
   "https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/"
  ],
  "notes": "Meta documents an explicit robots.txt bypass for the user-triggered fetcher, on the grounds the user requested it. Two tokens are commonly missing from third-party lists: 'meta-webindexer' (Meta AI search quality) and 'meta-externalads'. Meta also notes robots.txt changes take up to 24 hours to take effect.",
  "quotes": [
   {
    "text": "This crawler may bypass robots.txt rules",
    "url": "https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/"
   },
   {
    "text": "it performs fetches that were requested by the user",
    "url": "https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/"
   }
  ],
  "tokens": [
   {
    "token": "meta-externalagent",
    "kind": "training",
    "role": "Bulk crawler for training foundation AI models and direct content indexing."
   },
   {
    "token": "meta-webindexer",
    "kind": "search",
    "role": "Crawls the web to improve Meta AI search result quality."
   },
   {
    "token": "meta-externalfetcher",
    "kind": "user",
    "role": "Fetches individual links at a user's request to support AI task completion."
   },
   {
    "token": "facebookexternalhit",
    "kind": "other",
    "role": "Fetches link-preview metadata when content is shared on Meta apps."
   },
   {
    "token": "meta-externalads",
    "kind": "other",
    "role": "Crawls for advertising and business products."
   }
  ],
  "gap": {
   "bulk": [
    "meta-externalagent"
   ],
   "live": [
    "meta-webindexer",
    "meta-externalfetcher"
   ],
   "user": [
    "meta-externalfetcher"
   ],
   "headline": true
  }
 },
 "amazon": {
  "docs": [
   "https://developer.amazon.com/amazonbot"
  ],
  "notes": "Amazon has quietly expanded from one token to three. Amazonbot and Amzn-SearchBot honor robots.txt (Amzn-SearchBot falls back to the rules given to other search bots if not named explicitly); Amzn-User carries the standard user-initiated carve-out. Amazonbot caches robots.txt for up to 30 days, so changes are slow to take effect. A token list that only names 'Amazonbot' is now incomplete.",
  "quotes": [
   {
    "text": "may not follow all robots.txt directives",
    "url": "https://developer.amazon.com/amazonbot"
   },
   {
    "text": "supports user actions, such as responding to Alexa queries that require up-to-date information",
    "url": "https://developer.amazon.com/amazonbot"
   }
  ],
  "tokens": [
   {
    "token": "Amazonbot",
    "kind": "training",
    "role": "General crawler; content may be used to train Amazon AI models."
   },
   {
    "token": "Amzn-SearchBot",
    "kind": "search",
    "role": "Crawls to improve search experiences in Amazon products and services."
   },
   {
    "token": "Amzn-User",
    "kind": "user",
    "role": "User-triggered fetch, e.g. answering an Alexa query needing current information."
   }
  ],
  "gap": {
   "bulk": [
    "Amazonbot"
   ],
   "live": [
    "Amzn-SearchBot",
    "Amzn-User"
   ],
   "user": [
    "Amzn-User"
   ],
   "headline": true
  }
 },
 "mistral": {
  "docs": [
   "https://docs.mistral.ai/robots"
  ],
  "notes": "Mistral now documents THREE tokens, not one. A list naming only 'MistralAI-User' is outdated as of 2026-09-02. Mistral does not publish an explicit robots.txt-bypass carve-out for MistralAI-User; the doc frames all three as robots.txt-addressable user agents, with a separate IP list at https://mistral.ai/mistralai-user-ips.json for firewall enforcement. UA strings all take the form '...compatible; <Token>/1.0; +https://docs.mistral.ai/robots)'.",
  "quotes": [
   {
    "text": "Content crawled by MistralAI-Index is not used for generative AI training of any kind.",
    "url": "https://docs.mistral.ai/robots"
   }
  ],
  "tokens": [
   {
    "token": "MistralAI-Training",
    "kind": "training",
    "role": "Collects web content to build training datasets for Mistral's generative AI models."
   },
   {
    "token": "MistralAI-Index",
    "kind": "search",
    "role": "Automated crawling for Mistral search indexing; explicitly not used for training."
   },
   {
    "token": "MistralAI-User",
    "kind": "user",
    "role": "User-triggered fetch when a user's question requires visiting a page."
   }
  ],
  "gap": {
   "bulk": [
    "MistralAI-Training"
   ],
   "live": [
    "MistralAI-Index",
    "MistralAI-User"
   ],
   "user": [
    "MistralAI-User"
   ],
   "headline": true
  }
 },
 "bytedance": {
  "docs": [],
  "notes": "NO vendor documentation could be opened. The reference URL embedded in the user-agent string (zhanzhang.toutiao.com) is not reachable from outside China, and ByteDance publishes no English-language crawler policy, IP list, or robots.txt statement. The token 'Bytespider' is therefore attested only from server logs and third-party trackers, not from a vendor page. Multiple independent log analyses report it fetching robots.txt and then crawling disallowed URLs anyway - unverified, but it means robots.txt cannot be relied on here.",
  "quotes": [],
  "tokens": [
   {
    "token": "Bytespider",
    "kind": "training",
    "role": "ByteDance crawler collecting web data for search and recommendation/AI use."
   }
  ],
  "gap": null
 },
 "common_crawl": {
  "docs": [
   "https://commoncrawl.org/ccbot"
  ],
  "notes": "Full UA string documented as CCBot/2.0 (https://commoncrawl.org/faq/). Honors robots.txt via User-agent: CCBot. Common Crawl warns that other crawlers impersonate CCBot and publishes verification IP ranges at https://index.commoncrawl.org/ccbot.json. No user-triggered agent exists.",
  "quotes": [
   {
    "text": "CCBot/2.0 (https://commoncrawl.org/faq/)",
    "url": "https://commoncrawl.org/ccbot"
   }
  ],
  "tokens": [
   {
    "token": "CCBot",
    "kind": "training",
    "role": "Builds the public Common Crawl archive, which is widely reused as LLM training data."
   }
  ],
  "gap": null
 },
 "ai2": {
  "docs": [
   "https://allenai.org/crawler"
  ],
  "notes": "Full UA string: 'Mozilla/5.0 (compatible) AI2Bot (+https://www.allenai.org/crawler)'. The page offers the token so sites can filter or reject the traffic, but it does NOT contain an explicit robots.txt compliance commitment. The variant token 'ai2bot-dolma' seen in third-party lists is not on the current AI2 page.",
  "quotes": [
   {
    "text": "can be used to filter or reject traffic from our crawler if desired",
    "url": "https://allenai.org/crawler"
   }
  ],
  "tokens": [
   {
    "token": "AI2Bot",
    "kind": "training",
    "role": "Crawls web domains to collect content for training AI2's open language models."
   }
  ],
  "gap": null
 },
 "duckduckgo": {
  "docs": [
   "https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot"
  ],
  "notes": "Full UA string documented as 'DuckAssistBot/1.2; (+http://duckduckgo.com/duckassistbot.html)' - version has moved past 1.1. It is real-time/on-demand but DuckDuckGo does NOT claim a robots.txt bypass; the doc gives robots.txt as the supported opt-out, with a 72-hour propagation delay, and states opting out does not affect organic rankings or search inclusion.",
  "quotes": [
   {
    "text": "This data is not used in any way to train AI models.",
    "url": "https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot"
   },
   {
    "text": "the change will take effect after 72 hours and DuckAssistBot will stop crawling",
    "url": "https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot"
   }
  ],
  "tokens": [
   {
    "token": "DuckAssistBot",
    "kind": "search",
    "role": "Real-time crawl of pages for DuckDuckGo's AI-assisted answers; not used to train models."
   }
  ],
  "gap": null
 }
}