Free Tool

Robots.txt Generator

Generate a properly formatted robots.txt file for your website. Control how Googlebot, Bingbot, AI crawlers and other bots access your pages.

Site Settings

Used to auto-generate the sitemap URL
Choose "Custom" to set rules per user-agent below

User-Agent Rules

Sitemap

Auto-filled from site URL. Edit if your sitemap is at a different location.

Generated robots.txt

# robots.txt generated by Spacemen Digital # https://spacemendigital.com/tools/robots-txt-generator/ User-agent: * Allow: / Sitemap: https://example.com/sitemap.xml
How It Works

How to Use This Tool

1

Enter Your Site URL

Add your website URL and choose whether to allow all crawlers, block all crawlers, or set custom rules per bot.

2

Configure Crawl Rules

Add specific rules for Googlebot, Bingbot, AI crawlers like GPTBot and ChatGPT-User, and other search engine bots.

3

Download & Upload

Copy the generated code or download the robots.txt file, then upload it to the root directory of your website.

More Tools

Related Tools

Meta Tag Generator

Generate SEO-ready title, description, OG and Twitter meta tags.

Schema Markup Generator

Build valid JSON-LD structured data for rich results.

SERP Preview

See exactly how your page appears in Google search results.

Word Counter

Count words, characters, sentences and get keyword density analysis.

Need Help With Your Technical SEO?

Robots.txt is just one piece of the puzzle. Get a free technical SEO audit and ensure search engines can properly crawl and index your site.

Get Free Audit View Case Studies