Test Robots.txt Rules Online

Paste robots.txt and test whether a URL path is allowed or blocked for a given crawler. Does not fetch live files.

Paste only — cannot fetch live URLs.The robots.txt tester analyzes content you paste into the box. Browsers block websites from loading other sites' HTML (CORS), so we cannot enter a URL and pull your page automatically. Copy HTML from View Source (Ctrl+U / Cmd+Option+U), your CMS, or a saved file.
—Result
Paste robots.txt to testMatched rule

How to use

  1. Copy your robots.txt from yoursite.com/robots.txt or your CMS.
  2. Paste it into the robots.txt box.
  3. Enter a URL path and User-agent, then check Allowed or Blocked.

Frequently asked questions

Can this fetch my live robots.txt?

No. Paste the contents of your robots.txt file. Browsers cannot fetch files from arbitrary domains due to CORS.

How do I get my robots.txt?

Visit yoursite.com/robots.txt in a browser and copy the text, or copy from your server/CMS.

Which crawler rules are tested?

Enter a User-agent (e.g. Googlebot or *). The tester matches rules for that agent and falls back to * rules.

Does this match Google exactly?

It uses longest-match Allow/Disallow logic similar to Google. Edge cases may differ — verify in Google Search Console for production sites.

Last updated

Powered by maratool