AI Crawlers

What is Google-Extended?

Answer

Google-Extended is the robots.txt user agent that controls whether Google can use a site's content for AI training (Gemini, AI Overviews training). It is distinct from Googlebot, which controls Search indexing. Blocking Google-Extended does not affect organic Search rankings but does exclude the site from AI Overview citation eligibility.

What it controls

Google's AI products (Gemini, AI Overviews, future generative experiences) sample training and retrieval data subject to Google-Extended. Setting Disallow on Google-Extended opts the site out of AI use but leaves traditional Search indexing intact.

Should you block it

Almost never for AEO purposes. Blocking Google-Extended makes the site ineligible for AI Overview citations and Gemini training, dramatically reducing AI visibility. The only reason to block is a strict no-AI content policy.

How to verify

Audit robots.txt for any Google-Extended Disallow lines. The AI Agent Readiness Check tests for this and reports findings.

Want help shipping AEO into your site?

Run the free 50-signal AI Agent Readiness Check or book a free scoping call.

Score my site