Robots.txt Explained: Controlling How Google Crawls Your Site

The robots.txt file tells search engine crawlers which pages they can or cannot request from your site. It is located at yourdomain.com/robots.txt.

Key Directives

• User-agent: Specifies which crawler the rule applies to (e.g., Googlebot or * for all).
• Disallow: Tells the crawler NOT to visit a specific path (e.g., /admin/).
• Allow: Explicitly allows a path (useful to override a parent Disallow).
• Sitemap: Points the crawler to your XML sitemap.

Create Yours

Don't risk syntax errors. Use our Robots.txt Generator to build a valid file in seconds.