How-To

How to allow AI crawlers in robots.txt?

Answer

Add explicit 'User-agent: <bot>' blocks followed by 'Allow: /' for each major AI crawler: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, Bytespider, Meta-ExternalAgent and CCBot. End the file with a Sitemap directive pointing to your XML sitemap.

Example robots.txt

User-agent: GPTBot / Allow: / / User-agent: ClaudeBot / Allow: / / ... and so on for each AI crawler. Then User-agent: * / Allow: / / Sitemap: https://example.com/sitemap.xml.

Common mistakes

Inheriting overly aggressive bot rules from a CDN config or WAF that block GPTBot accidentally. Forgetting Google-Extended (it is distinct from Googlebot). Not declaring the sitemap in robots.txt.

How to verify

Use the AI Agent Readiness Check which tests crawler access for all major AI bots and reports any blocks.

Related questions

Should I allow GPTBot in robots.txt?→What is GPTBot?→What is ClaudeBot?→What is Google-Extended?→

Want help shipping AEO into your site?

Run the free 50-signal AI Agent Readiness Check or book a free scoping call.

Score my site