robots.txt Generator & Tester
Write a robots.txt from paths to block or full syntax — agent groups, Allow/Disallow with * and $, Sitemap — and test whether a URL is allowed (RFC 9309).
A clean robots.txt, the parsed groups and sitemaps, a verdict for a test URL and agent naming the winning rule and group, lint notes (paths without /, Crawl-delay, unknown fields) and the file size.
Example: Disallow /admin/ and /tmp/*.pdf$ with Allow /admin/public/: /admin/public/x is allowed for Googlebot by the longer Allow rule; /tmp/a/b.pdf is blocked, /tmp/a.pdfx is not (the $ anchor).
RFC 9309 rules,
with a tester built in.
How the file is built, how the tester decides, and what the lint notes mean.
Writing the file
In the simple form each line is a path to disallow for the user-agent you name, with allow paths, a sitemap and a crawl delay from the fields. Paste full robots.txt syntax instead and it is parsed and rewritten: consecutive User-agent lines share one group, Allow/Disallow lines belong to the group above them, Sitemap lines are global, comments are dropped. Empty Disallow means allow everything.
The tester
RFC 9309 (the Robots Exclusion Protocol, as Google, Bing and others implement it): the crawler picks the group whose user-agent token is the longest prefix of its name, falling back to *; within that group the rule with the longest matching path wins, and Allow wins a tie; * matches any characters and $ anchors the end of the path. A URL is reduced to its path and query before matching. The verdict names the rule and the group that decided it.
Lint and limits
Paths should start with /; Crawl-delay is honoured by Bing and Yandex but ignored by Google; Noindex, Host and other non-standard fields are flagged; files over 500 KiB are truncated by Google. robots.txt controls crawling only — it does not stop indexing of URLs discovered elsewhere and it is not access control. Nothing leaves the browser; the same four anonymous usage counts as the rest of the site apply.
SOURCES
- RFC 9309 Robots Exclusion Protocol: group selection by the most specific user-agent token, longest-match rule with Allow winning ties, * and $ in paths, 500 KiB parse limit
Last reviewed 22 September 2026. How results are checked: How we verify.