What a robots.txt file does, and what it does not
A robots.txt file is a plain text file at the root of a host, such as https://example.com/robots.txt, that tells crawlers which URL paths they may request. The format is standardised as the Robots Exclusion Protocol in RFC 9309. It controls crawling, not indexing. A URL blocked in robots.txt can still appear in search results, usually with no description, if other pages link to it. Worse, a crawler that cannot fetch the page cannot see a noindex tag on it. To keep a page out of search, leave it crawlable and add noindex, or put it behind a login. Robots.txt is also not a security measure: it is public, and it advertises the very paths you list.
What this generator outputs
The tool always starts with a group for User-agent: * (all crawlers). Choose "Allow all (with exceptions)" and each line in the disallow box becomes a Disallow: line, or choose "Block all crawlers" for Disallow: /. Each line in the allow box becomes an Allow: line in the same group, which is how you open a sub-path inside a blocked folder. With both boxes empty you get a bare Disallow:, which means nothing is blocked. A crawl-delay value adds a Crawl-delay: line to that group. Bots named in the "Block specific bots entirely" field (for example GPTBot, CCBot) each get their own group with Disallow: /, and a Sitemap URL is added as a final Sitemap: line.
For anything finer, press "Add bot group". Each group takes one or more bot names (comma separated, for example Googlebot, Bingbot, written as consecutive User-agent: lines), its own Disallow and Allow paths and its own crawl-delay. A crawler obeys only the single most specific group that names it and ignores the * group, so a group for Googlebot must repeat any rule from the general group that Googlebot should still follow. The defaults produce:
User-agent: *
Disallow: /admin
Disallow: /cart
Disallow: /checkout
Sitemap: https://example.com/sitemap.xmlThe input is checked before it is written. A disallow entry that does not start with / (or *), such as a pasted full URL, is left out with a warning, because crawlers would not match it as intended. The same check applies to Allow paths and to every bot group. Crawl-delay must be a plain number of seconds, bot names may only use letters, digits, dots, dashes and underscores (or *), and the Sitemap must be a full http(s) URL; anything invalid is omitted and explained. The tool also warns when one bot appears in two groups, because crawlers merge those groups, which is easy to misjudge.
How matching works (RFC 9309)
| Rule | Behaviour |
|---|---|
Disallow: /admin | Prefix match. Blocks /admin, /admin/users and /administrator. |
Disallow: /admin/ | Blocks only paths inside that folder. |
| Case | Paths are case-sensitive: /Admin is a different path. |
| Wildcards | * matches any characters and $ anchors the end, for example Disallow: /*.pdf$. |
| Conflicts | The longest matching rule wins; if an Allow and Disallow tie, Allow wins. |
| Missing file (4xx) | Crawlers treat everything as allowed. |
| Server error (5xx) | Crawlers may assume everything is disallowed until it recovers. |
Common mistakes
- Shipping "Block all crawlers" to production. It is right for a staging site and disastrous on a live one. Check
/robots.txtafter every launch. - Blocking CSS and JavaScript. Google renders pages, and blocked resources can make a page look broken to it.
- Expecting Crawl-delay to work everywhere. It is not part of RFC 9309. Google ignores it, though some other crawlers, including Bing's, honour it.
- A relative Sitemap URL. The Sitemap line is an extension supported by the major engines and needs a full absolute URL.
- Wrong location. The file only applies to the exact protocol, host and port it is served from. A copy in a subfolder is ignored.
After uploading, test specific URLs with the robots.txt tester, and build the file it points to with the XML sitemap generator. The meta tag generator can add a noindex tag when you need to keep a page out of results.