What is Web Bot Auth?
Web Bot Auth is a proposed standard that lets a bot or AI agent prove who it is by signing each HTTP request with a private key. Websites and CDNs check the signature against public keys the operator publishes, instead of trusting the user-agent string alone.
Our Web Bot Auth guide shows the request headers and key directory, and which CDNs and AI companies support it so far.
Updated
Guides that cover this
Web Bot Auth explained: headers, keys, who supports it
7 min readWeb Bot Auth is a proposed standard that lets a bot or AI agent prove who it is by cryptographically signing each HTTP request, using HTTP Message Signatures (RFC 9421). The bot publishes its public keys at a well-known URL, and the website or CDN checks each signature against those keys, so a request that claims to come from a known agent can be verified instead of trusted on its user-agent string alone. Cloudflare proposed it in 2025 and the work has since moved to the IETF.
AI training in robots.txt: __usage_ai-training not live yet
8 min readGoogle says it will read user-agent: __usage_ai-training from early 2027, but it does nothing today. There is no single "AI training" switch in robots.txt. You opt out of training by naming each company's training crawler or token (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot and MistralAI-Training) with Disallow: /. You stay in AI answers by leaving the search crawlers allowed: OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot.