Web Bot Auth explained: headers, keys, who supports it

A user agent is only a claim#

Picture your access log on a busy morning. A few hundred requests say they come from GPTBot. You allowed GPTBot on purpose, because you followed our guide to allowing it in robots.txt and you want to show up in ChatGPT answers. Now a question you can't answer from the log alone: are those requests really from OpenAI?

The user agent string is a line of text the client chooses to send. Any scraper can type "GPTBot" into it, which means the name tells you what the client wants you to believe and nothing more.

The usual fixes are reverse DNS lookups and published IP ranges. OpenAI, Google and others publish the addresses their crawlers use, and you can check each request against them. That works, but it is brittle. Ranges change, lists go stale, and it breaks down badly for the newer kind of traffic: AI agents that browse on behalf of a person, often from shared cloud infrastructure where an IP address says very little about who is behind it.

Web Bot Auth tries to fix this by moving identity out of the network layer and into the request itself.

How does Web Bot Auth work?#

The idea is the same one that sits behind signed software releases. The bot operator creates a key pair (the drafts use Ed25519). The private key stays with the operator. The public key goes into a small JSON file, a JWK Set, served at a fixed path on the operator's own domain:

/.well-known/http-message-signatures-directory

Every request the bot sends then carries three extra headers. Here is an illustrative example of what they look like. The key ID and the signature are made-up values, and the domain is a placeholder. Signature-Agent syntax also differs between sources (a bare hostname in Cloudflare's post, a quoted URL elsewhere), so don't copy this into a parser and check the spec version you are implementing:

Signature-Agent: "https://bot.example.com"
Signature-Input: sig1=("@authority" "signature-agent");created=1762000000;expires=1762000300;keyid="poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U";alg="ed25519";tag="web-bot-auth"
Signature: sig1=:jdq0SqOwHdyHr9+r5jw3iYZH6aNGKijYp/EstF4RQTQdi5N5YYKrD+mCT1HA1nZDsi6nJKuHxUi/5Syp3rLWBA==:

Signature-Agent points to where the keys live. Signature-Input lists which parts of the request were signed (here the host being visited and the Signature-Agent header itself), when the signature was made, when it expires, and which key signed it. The tag="web-bot-auth" marks the purpose so the signature can't be quietly reused for something else. Signature is the signature itself.

The other half is the key directory the Signature-Agent header points to. It is a JWK Set served over HTTPS, and for Ed25519 each key is an OKP entry with the public part in x. The keyid in the header is the base64url-encoded SHA-256 thumbprint of that JWK, which follows RFC 7638 and RFC 8037. The directory draft (an individual draft) defines the JWKS directory and its media type, application/http-message-signatures-directory+json. The key below is a placeholder, so it won't match the made-up keyid above:

{"keys": [{"kty": "OKP", "crv": "Ed25519", "x": "PLACEHOLDER_BASE64URL_PUBLIC_KEY"}]}

Because the signature covers the host and has a short expiry, someone who captures one signed request can't replay it against a different site or use it next week.

What the server checks#

Most site owners will never write this code, because their CDN or WAF does it for them. It still helps to know what "verified" actually means. A server that supports Web Bot Auth does roughly this, in order:

1. Confirm the Signature-Input entry has tag="web-bot-auth" and that created and expires are within an acceptable window.
2. Read the Signature-Agent URL and fetch the key directory from its /.well-known/http-message-signatures-directory path (caching it, not fetching on every request).
3. Find the public key whose thumbprint matches keyid.
4. Rebuild the signature base from the covered components and verify the signature with that key.
5. Map the verified operator to a policy: allow, rate limit, or block.

The last step is the one that matters for you. Verification only tells you who is asking. What you let them do is still your decision.

Web Bot Auth vs IP lists and user-agent strings#

	User-agent string	Published IP ranges	Web Bot Auth
Can it be faked?	Yes, trivially	Hard, but lists go stale	Not without the private key
Works for agents on shared cloud IPs	No	Poorly	Yes
Who maintains it	Nobody	Each operator, each site checks lists	Operator hosts keys, CDN verifies
Survives the operator changing networks	Yes	No, until lists update	Yes
Effort for a site owner	None	Scripts or WAF rules	Usually a CDN setting

I wouldn't throw away IP checks yet. Plenty of crawlers don't sign anything, so for now Web Bot Auth sits alongside the older methods rather than replacing them.

Who supports Web Bot Auth so far?#

Cloudflare introduced Web Bot Auth in May 2025 and uses it in its verified bots and signed agents programs, so a bot that signs its requests can be recognised by any site behind Cloudflare's bot management. According to third-party write-ups, OpenAI's Operator signs its requests. The sources I found don't show GPTBot or OAI-SearchBot signing, so keep verifying them by their published IP ranges, as described in Cloudflare's docs and OpenAI's bots page.

Other vendors have followed. AWS WAF Bot Control added Web Bot Auth verification in November 2025 for WAF on CloudFront distributions. Verified traffic is allowed by default. Shopify lets a merchant generate signed credentials in the admin for their own crawlers and tools, so those requests are recognised on their store. That covers a merchant authorising their own tools for their own store, not third-party bots. Akamai says its bot and abuse products support Web Bot Auth, and lets customers allow, challenge, delay or block verified traffic. Visa's Trusted Agent Protocol for agent-driven payments builds on the same signature approach. Vercel's bot verification now accepts Web Bot Auth alongside its IP and reverse DNS checks, so a verified bot that signs its requests can get through Vercel's Bot Protection and Challenge Mode, as its changelog describes.

Google is the cautious one. Its Web Bot Auth page calls the protocol experimental and says it is testing it with some AI agents hosted on Google infrastructure, so not all Google user agents sign their requests. Google still advises relying on IP addresses, reverse DNS and user-agent strings while it gradually rolls out signed traffic. That means you can't treat a missing signature from a Google bot as a red flag yet.

On the standards side, the IETF has an active Web Bot Auth working group (webbotauth). The architecture draft began as an individual submission by Cloudflare's Thibault Meunier. The standard is still in progress, so vendors may implement different draft versions and details can differ between them. The working group's charter covers standards-track techniques for authenticating automated clients to websites meant for humans, a way to convey more information about a requesting bot, and operational guidance. The directory document is still an individual Internet-Draft, not an RFC. There is also a real open question about openness. The charter asks the group to address impacts on the openness of the web, and at IETF 126 (July 2026) the group also heard an anonymous-authentication proposal, as the meeting minutes record. If you want to try it, Cloudflare's open source web-bot-auth repository on GitHub has libraries for signing and verifying, plus a browser extension that signs requests so you can test your own setup.

Does it change robots.txt?#

No, and this is the mistake I'd most like to head off. Web Bot Auth answers "who is this?". robots.txt answers "what do I want this bot to do?". They are separate layers.

A signed request is not a promise to obey your rules. It does make enforcement much easier, though. If you've decided to block AI training but stay in AI search answers, signatures let your CDN apply that decision to the real operator and drop impostors using the same name. Your robots.txt groups for GPTBot, ClaudeBot and the rest stay exactly as they are. The AI crawlers list is still where to get those names right.

What I'd do now#

If you run a website, nothing breaks if you ignore Web Bot Auth today. Unsigned bots keep working exactly as before. I'd still start with two small things. If you are behind Cloudflare or another CDN that supports it, look at your bot settings and see whether signed agents are already being recognised and how they are treated. For Google's own crawlers and agents, keep checking published IP ranges, reverse DNS and user agents, because Google's experimental support is limited and it still advises those checks. Then add the Signature-Agent header to your access logs, because that single field will tell you which agents are already signing their traffic to your site.

If you run a crawler or an AI agent, signing is the cheapest way to stop being blocked as "probably a scraper". Generate a key, publish the directory, sign with an existing library instead of writing your own crypto, and register with the CDNs your users' targets sit behind.