Quick Presets
Block Paths
Custom Rules
Robots.txt Best Practices
Location: Upload robots.txt to your root domain (e.g., yoursite.com/robots.txt). It must be at the root level.
Not security: Robots.txt is a suggestion, not a security measure. Don't use it to hide sensitive content - use authentication instead.
Test first: Use Google Search Console's robots.txt tester before deploying. Mistakes can deindex your entire site.
Sitemap: Always include your sitemap URL. This helps search engines discover all your important pages.