SmartToolsNest
All 188 ToolsAudio Tools6Developer Tools25PDF Tools20Image Tools20Text Tools9SEO Tools54Domain Tools15Business Tools38Document Tools1
Developer ToolSEO Tool

Robots.txt Tester & Checker

Test whether Googlebot or another crawler can access a public URL and see the matched rule.

Googlebot accessMatched ruleCrawl verdict

1. Paste a public URLUse the exact page address, including https://.

2. Run the checkOnly information needed for this tool is inspected.

3. Follow the evidenceFix confirmed blockers before lower-priority notes.

Only public HTTP/HTTPS URLs on ports 80 and 443 are accepted. Private networks are blocked.

How to Use

Enter a public URL or page details, run the check, and follow the plain-language result guidance.

Secure Inspection

Private networks, unsafe ports, oversized responses, excessive redirects, and abusive request rates are blocked.

Important

Scores explain observable signals and are not ranking, safety, or availability guarantees.

Practical guide

Robots.txt Tester: Check Googlebot URL Access

Use this free robots.txt tester to check whether Googlebot or another crawler is allowed to access a public page path. The checker fetches the live robots.txt file, evaluates the URL against its Allow and Disallow rules, and shows the exact matched evidence behind the crawl verdict.

How to use this tool

  1. 1Enter the complete public URL of the page you want to test.
  2. 2Run the inspection to fetch that hostname’s robots.txt file.
  3. 3Review the crawl verdict and exact matched allow or disallow rule.
  4. 4Update the live file if needed, then test the public URL again.

Example rule match

Page: https://example.com/admin/report
Rule: Disallow: /admin/
Result: blocked for the matching crawler group

What the result means

  • A crawl verdict does not prove whether a URL is indexed.
  • Rules are hostname- and protocol-specific; test the exact live URL.
  • Crawler implementations can differ, so verify critical changes in the relevant search-engine tools.

Frequently asked questions

What does allowed by robots.txt mean?+

It means the tested rules do not block Googlebot or the selected crawler from requesting the path. It does not guarantee crawling, indexing or ranking.

Can I use this as a Googlebot robots.txt checker?+

Yes. Test the exact public URL and review the matched user-agent group and winning rule shown in the result.

Why can a blocked URL still appear in search?+

Search engines may discover the URL from links even when they cannot crawl its content. Robots.txt is not a guaranteed removal mechanism.

Does the tester access private network URLs?+

No. The server-side fetcher accepts public HTTP or HTTPS destinations and blocks private networks and unsafe ports.

Developer newsletter

Stay ahead with new tools

Get practical developer guides, new tools, and workflow improvements without inbox noise.