Robots.txt Generator

Quick Presets

Sitemap URL

Block Paths

WordPress Admin

/wp-admin/

WP Includes

/wp-includes/

Shopping Cart

/cart/

Checkout

/checkout/

User Account

/my-account/

Search Results

/*?s=

CGI Scripts

/cgi-bin/

Private Folder

/private/

Custom Rules

robots.txt

Robots.txt Best Practices

Location: Upload robots.txt to your root domain (e.g., yoursite.com/robots.txt). It must be at the root level.

Not security: Robots.txt is a suggestion, not a security measure. Don't use it to hide sensitive content - use authentication instead.

Test first: Use Google Search Console's robots.txt tester before deploying. Mistakes can deindex your entire site.

Sitemap: Always include your sitemap URL. This helps search engines discover all your important pages.

Start with free credits.
Skip the subscription.

This free tool is one slice of the job. The full kit - rank tracking, audits, competitor gaps, content, and your visibility across ChatGPT and Google's AI Overviews - runs on credits you only spend when you use them. Sign up and we'll drop 25 credits in your account to try the paid tools on us.