HandyBench

Robots.txt Tester

Test any path against a live robots.txt, rule by rule.

This check runs through this site's own server, so it works for any public URL — no CORS or browser restrictions.

How to use the Robots.txt Tester

  1. 1 Enter a domain and click Fetch robots.txt.
  2. 2 Enter a URL path (e.g. /private/page) and pick a user-agent.
  3. 3 Read the ALLOWED or BLOCKED verdict and the exact matching Allow or Disallow rule.

How the tester decides allowed or blocked

The tool fetches /robots.txt from the root of the host you enter, then applies the matching rules from RFC 9309 (the Robots Exclusion Protocol) in your browser, so changing the path or user-agent gives an instant answer without another fetch.

  1. Parse. Comments (# to end of line) are stripped. Consecutive User-agent lines form one group, and the Allow and Disallow lines after them belong to that group. Sitemap lines are collected separately.
  2. Pick a group. Every group naming your selected crawler is used, merged into one rule set if the file repeats the user-agent (RFC 9309). Googlebot-Image, Googlebot-News and Googlebot-Video use a Googlebot group if they have no group of their own. If nothing names the crawler, the * group is used. If there is no * group either, everything is allowed.
  3. Match patterns. Percent-encoding is normalised first on both the path and the patterns (non-ASCII becomes UTF-8 %XX, %7E becomes ~, other triples are upper-cased), so equivalent spellings match. A pattern matches from the start of the path. * matches any run of characters and a trailing $ anchors the end. Every other character, including ? and ., is matched literally, so Disallow: /search?q= blocks /search?q=shoes and Disallow: /*? blocks any URL with a query string. An empty Disallow: blocks nothing.
  4. Longest match wins. The rule with the longest pattern (not counting the $) is applied. If an Allow and a Disallow tie on length, Allow wins.

Worked example

Suppose the file contains this group for *:

User-agent: *
Disallow: /private/
Allow: /private/press/
Disallow: /*.pdf$
Allow: /page
Disallow: /page
Path testedVerdictWhy
/private/report.htmlBlockedDisallow: /private/ is the only match (9 characters)
/private/press/kitAllowedAllow: /private/press/ (15) beats Disallow: /private/ (9)
/files/a.pdfBlockedmatches /*.pdf$
/files/a.pdf?x=1Allowedthe $ anchor fails because of the query string
/pageAllowedAllow and Disallow tie at 5 characters, so Allow wins

Where this tester is simpler than a real crawler

  • A fixed list of user-agents. You can pick Googlebot, Googlebot-Image, Googlebot-News, Googlebot-Video, Bingbot, GPTBot or *. Other crawlers with special handling, such as Google's AdsBot (which ignores * groups), are not modelled.
  • The path box is forgiving. You can enter a path, a path with a query string, or a full URL; only the path and query are tested, and a missing leading slash is added.
  • Status handling. A found file requires HTTP 200. A 4xx (such as 404) is shown as "not found" and treated as allow-all. A 5xx (and 429, as Google treats it) means the file is unreachable, and RFC 9309 says crawlers must then assume everything is disallowed, so the tester shows BLOCKED with that explanation. Google falls back to allow-all if the errors last more than about 30 days. Either way a 5xx is a problem to fix.
  • Only the first 200 KB of the file is read, and the file is fetched from the host you type, over https. Redirects are followed for up to 10 hops, and every hop is checked against the same safety rules as the first request (a redirect to a private or internal address is refused with "Redirect blocked"). The URL shown above the file is the final one after redirects, so a request for example.com that lands on www.example.com shows the www address. www and non-www can serve different files.

Disallow controls crawling, not indexing. To write a file, use the robots.txt generator, and list your sitemap with the XML sitemap generator.

Examples

  • ▸ With Disallow: /private/ and Allow: /private/press/, the path /private/press/kit is ALLOWED (the 15-character Allow beats the 9-character Disallow) but /private/report.html is BLOCKED.
  • ▸ A rule Disallow: /*.pdf$ blocks /files/a.pdf but not /files/a.pdf?x=1, because the $ end anchor no longer matches once a query string is added.
  • ▸ When an Allow and a Disallow of equal length both match, such as Allow: /page and Disallow: /page, the path is ALLOWED.

Frequently asked questions

How does robots.txt rule matching actually work?

A crawler uses the group that names it, falling back to its parent product (Googlebot-Image uses Googlebot) and then to "*". Groups that name the same crawler are merged. Within the chosen set, the longest matching Allow or Disallow pattern wins; if an Allow and a Disallow match with equal length, the Allow wins.

Why test against a specific bot like GPTBot instead of just "*"?

Many sites now write separate rules for AI crawlers (GPTBot, CCBot, etc.) that differ from their rules for search engines like Googlebot — testing per user-agent shows you exactly what each one is allowed to fetch.

Does blocking a page in robots.txt guarantee it won't be indexed?

No. Disallow only stops crawling — a blocked URL can still appear in search results (often with no snippet) if it is linked from elsewhere. Use a noindex meta tag or header if you need a page kept out of the index entirely.

What does a Sitemap: line in robots.txt do?

It points crawlers to your XML sitemap URL so they can discover pages more efficiently. A robots.txt file can list multiple Sitemap: lines, and this tool surfaces every one it finds.

Does the tester follow redirects, and which URL does it show?

Yes, up to 10 hops (RFC 9309 says crawlers should follow at least 5), which covers the common http to https and non-www to www redirects. Each hop is safety-checked, so a redirect to a private or internal address is refused. The URL shown above the file is the final one after redirects, because that is the file crawlers would actually read.

How are ? and other special characters in a rule handled?

Only * (any sequence of characters) and a trailing $ (end of URL) are special. Everything else, including ? and ., is literal, so Disallow: /search?q= blocks /search?q=shoes, and Disallow: /*? blocks any URL that has a query string.

What if the site has no robots.txt at all?

The tester reports the HTTP status and says the file was not found. Crawlers treat a missing robots.txt (a 404) as allow-all. A server error is different: for a 5xx response (and 429, as Google handles it) RFC 9309 says to assume everything is disallowed, and the tester shows the path as BLOCKED with that explanation. Google falls back to allow-all if the error lasts more than about 30 days.

Which crawlers can I test, and where does this tester differ from Google?

You can pick Googlebot, Googlebot-Image, Googlebot-News, Googlebot-Video, Bingbot, GPTBot or the * group. Repeated groups for one user-agent are merged and Google-specific bots fall back to Googlebot as RFC 9309 and Google describe. Percent-encoding is normalised, so /%7Euser and /~user match. Other special Google crawlers (such as AdsBot) have their own rules that are not modelled.

Related tools

SEO

Robots.txt Generator

Build a valid robots.txt in seconds.
SEO

XML Sitemap Generator

Turn a URL list into a valid sitemap.xml.
SEO

Website Meta Tag Checker

See exactly what meta tags, OG data and SERP snippet any page sends.