SmartToolsNest
All 188 ToolsAudio Tools6Developer Tools25PDF Tools20Image Tools20Text Tools9SEO Tools54Domain Tools15Business Tools38Document Tools1
Developer ToolSEO Tool

Free Robots.txt Generator

Create Allow, Disallow, User-agent and Sitemap rules for Googlebot and other crawlers.

Crawler controlsSitemap directiveCopy + download

1. Add detailsStart with the first field; examples appear inside empty fields.

2. See the resultThe preview updates automatically while you type.

3. Copy or downloadReview warnings before using the output.

Start here

Fill the simple fields below. Your result updates instantly.

Your result

Start typing above to see the result.


                        
Checks and warnings
  • Add content to begin.
Practical guide
Free Robots.txt Generator for SEO and AI Crawlers
Use this free robots.txt generator to build readable crawler rules without memorizing directive syntax. Choose Googlebot, all crawlers or another user-agent, enter Allow and Disallow paths, add your XML sitemap URL, and review the file before publishing it at your domain root.
How to use this tool
  1. 1Choose All crawlers unless a rule genuinely applies to one named bot.
  2. 2Enter directory or page paths beginning with a forward slash.
  3. 3Add an absolute sitemap URL, review the preview, then download robots.txt.
  4. 4Upload the file to your domain root and verify it from your public URL.
Example robots.txt file

                    
User-agent: *
Disallow: /admin/
Disallow: /internal-search/
Allow: /assets/
Sitemap: https://example.com/sitemap.xml

                
What the result means
  • Robots.txt controls crawling, not guaranteed indexing or removal from search.
  • Do not block CSS or JavaScript files needed to render public pages.
  • A disallowed URL can still appear in search if other pages link to it. Use an appropriate noindex method when access is allowed.
Frequently asked questions
Where should robots.txt be uploaded?+
Publish it at the root of the hostname, such as https://example.com/robots.txt. A file inside a subdirectory does not control the whole site.
Can I create rules for Googlebot and AI crawlers?+
Yes. Select a named user-agent when it genuinely needs different rules, and verify that crawler documentation before publishing a site-wide block.
Can robots.txt remove a page from Google?+
No. It manages crawler access. A blocked URL may still be discovered through links, so use the correct indexing or removal method for your goal.
Should every website include a sitemap directive?+
It is optional, but an absolute sitemap URL helps supported crawlers discover the sitemap location.
Developer newsletter
Stay ahead with new tools
Get practical developer guides, new tools, and workflow improvements without inbox noise.