Google Tests Web Bot Auth Protocol for Bot Verification
Configure your site to validate Web Bot Auth signatures using the Signature‑Agent header and the bot’s JWKS, while retaining IP/reverse DNS checks.
Configure your site to validate Web Bot Auth signatures using the Signature‑Agent header and the bot’s JWKS, while retaining IP/reverse DNS checks.
Summary
Google has begun testing Web Bot Auth, an experimental protocol that lets websites cryptographically verify automated traffic. Built on HTTP Message Signatures (RFC 9421), the protocol requires bots to sign each request with a private key and publish the public key in a JSON Web Key Set at a well‑known URL. Requests include a new Signature‑Agent header that points to the bot’s key directory, allowing the site to validate the signature against the published JWKS. The current rollout only signs a subset of Google‑hosted agents, such as the Google‑Agent fetcher, and the protocol is still experimental. Site owners are advised to keep using IP‑based checks, reverse DNS, and user‑agent strings as a fallback while the protocol matures. The protocol offers a more tamper‑proof way to distinguish legitimate crawlers from spoofed bots, but it does not automatically block traffic and requires hosting providers to support the standard.
Key changes
- Introduces Web Bot Auth protocol for bot verification
- Built on HTTP Message Signatures (RFC 9421)
- Bots sign requests with a private key and publish a JWKS at a .well‑known URL
- Requests carry a Signature‑Agent header pointing to the bot’s key directory
- Only a subset of Google‑hosted agents (e.g., Google‑Agent) are signed in the current rollout
- Protocol is experimental; sites should continue using IP, reverse DNS, and user‑agent checks
- Does not automatically block traffic; verification is optional
- Provides a tamper‑proof method to distinguish legitimate crawlers from spoofed bots