Web Bot Auth is an IETF draft standard for cryptographically authenticating bot requests using HTTP Message Signatures. Sites publish a directory at /.well-known/http-message-signatures-directory listing public keys for trusted bots. The standard solves the long-running problem of distinguishing legitimate AI crawlers from impersonators.
Sites have no reliable way to tell whether a request claiming to be GPTBot actually is GPTBot. Web Bot Auth lets bots sign their requests, and sites verify the signature against published keys.
Sites can distinguish authentic AI crawlers from impersonators and apply different policies (allow GPTBot, throttle unknown). Cleaner robots.txt compliance. Reduced scraping abuse.
Publish a key directory at /.well-known/http-message-signatures-directory. Verify request signatures server-side. The Spacemen Digital AI Agent Readiness Check tests for this directory as a frontier-standards check.
Run the free 50-signal AI Agent Readiness Check or book a free scoping call.
Score my site