Robots.txt Tester
Test any path against a live robots.txt, rule by rule.
This check runs through this site's own server, so it works for any public URL — no CORS or browser restrictions.
How to use the Robots.txt Tester
- 1 Enter a domain and click Fetch robots.txt.
- 2 Enter a URL path (e.g. /private/page) and pick a user-agent.
- 3 Read the ALLOWED or BLOCKED verdict and the exact matching Allow or Disallow rule.
Frequently asked questions
How does robots.txt rule matching actually work?
A crawler first picks the most specific user-agent group that names it exactly, falling back to the "*" group. Within that group, the longest matching Allow or Disallow pattern wins; if an Allow and a Disallow match with equal length, the Allow wins.
Why test against a specific bot like GPTBot instead of just "*"?
Many sites now write separate rules for AI crawlers (GPTBot, CCBot, etc.) that differ from their rules for search engines like Googlebot — testing per user-agent shows you exactly what each one is allowed to fetch.
Does blocking a page in robots.txt guarantee it won't be indexed?
No. Disallow only stops crawling — a blocked URL can still appear in search results (often with no snippet) if it is linked from elsewhere. Use a noindex meta tag or header if you need a page kept out of the index entirely.
What does a Sitemap: line in robots.txt do?
It points crawlers to your XML sitemap URL so they can discover pages more efficiently. A robots.txt file can list multiple Sitemap: lines, and this tool surfaces every one it finds.