A robots.txt file tells crawlers which parts of your site to leave alone. This generator writes a valid one from a few choices, free, with no account.
Quick Presets
Block Paths
Custom Rules
robots.txt
Robots.txt Best Practices
Location: Upload robots.txt to your root domain (e.g., yoursite.com/robots.txt). It must be at the root level.
Not security: Robots.txt is a suggestion, not a security measure. Don't use it to hide sensitive content - use authentication instead.
Test first: Use Google Search Console's robots.txt tester before deploying. Mistakes can deindex your entire site.
Sitemap: Always include your sitemap URL. This helps search engines discover all your important pages.
What This Generates
Pick the crawlers you want to allow or block, add any paths that should stay out of search, and copy the result. The file goes at the root of your domain, at /robots.txt, and nowhere else. A robots file in a subdirectory is ignored completely, which is the most common reason a carefully written one does nothing at all.
The syntax is small. User-agent names the crawler a rule applies to, Disallow blocks a path, Allow carves an exception out of a block, and an asterisk matches any crawler. Rules are grouped by user-agent, and a crawler obeys the most specific group that names it, ignoring the rest. That last detail catches people out constantly: if you write a block for Googlebot and a separate one for everyone, Googlebot reads only its own group and your general rules never apply to it.
Three things in that file are worth explaining. The /*?s= line blocks internal search result pages, which are the classic source of thousands of thin URLs that get crawled and never indexed. The exception for admin-ajax.php exists because blocking it breaks front-end functionality on a lot of WordPress sites while saving nothing. And the empty Disallow under GPTBot is not a mistake: an empty value allows everything, which is how you explicitly permit a crawler rather than leaving it to the general rule.
That last block is a decision, not a default. Blocking AI crawlers keeps your content out of training sets, and it also keeps you out of the answers those systems generate. If being cited by AI engines matters to your business, blocking them is working against yourself.
What It Cannot Do
Robots.txt is a crawl instruction, not an access control and not a way to remove a page from search. A blocked URL can still be indexed if other sites link to it, and it will appear in results with no description because the crawler was never allowed to read it. To keep a page out of the index, let it be crawled and serve a noindex directive. Blocking it in robots.txt prevents the crawler from ever seeing that directive, which produces the exact opposite of the intended result.
It is also public. Anyone can read yours, and listing a private directory in it advertises the directory. Real protection is authentication, not a text file that politely asks.
Questions
Do I need a robots.txt file at all?
No. A site with no robots.txt is crawled normally. You need one when you have specific paths worth keeping out of the crawl, not as a matter of routine.
Will blocking a page remove it from Google?
No, and this is the mistake that costs people. Use a noindex directive on the page itself, and leave the page crawlable so the directive can be read.
Should I block AI crawlers?
Only if you have a reason. Blocking GPTBot, ClaudeBot, and PerplexityBot removes your content from those systems, including from the answers they give about your industry.
Is this tool free?
Yes, with no account and no limit. It runs in your browser.
A robots.txt controls crawling on one site. An account adds the crawl that tells you what those rules are actually doing across every URL.
This free tool is one slice of the job. The full kit - rank tracking, audits,
competitor gaps, content, and your visibility across ChatGPT and Google's
AI Overviews - runs on credits you only spend when you use them. Sign up and we'll
drop 25 credits in your account to try the paid tools on us.
