AI Crawlers

Should I allow GPTBot in robots.txt?

Answer

For Answer Engine Optimization, yes. Allowing GPTBot is the default for AEO programs because it makes the site eligible for inclusion in ChatGPT training and live browse citations. Blocking GPTBot is only justified for sites with strict no-AI content policies or paywalls that explicitly exclude AI access.

The case for allowing

ChatGPT increasingly answers commercial queries directly. Brands cited by ChatGPT gain visibility and trust. Brands blocked from ChatGPT are functionally invisible to a fast-growing share of search traffic.

The case for blocking

Paywalled content (NYT-style), strict legal IP positions, or brands that intentionally want to be invisible to AI. These are minority cases. For most marketing-driven sites, blocking is self-defeating.

How to set it correctly

Add 'User-agent: GPTBot' followed by 'Allow: /' to your robots.txt. Do the same for ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, Bytespider and CCBot. Validate with the AI Agent Readiness Check.

Want help shipping AEO into your site?

Run the free 50-signal AI Agent Readiness Check or book a free scoping call.

Score my site