Robots.txt Rule Tester

Test a saved set of rules

Paste rules or open a UTF-8 robots.txt file. Enter its root-level URL and a crawler product token such as ExampleBot, not a full browser User-Agent header. Add absolute HTTP/HTTPS URLs or root-relative paths, one per line. Nothing is fetched from the supplied robots URL or tested websites.

Each result shows allowed, disallowed or unclassified, plus the winning source line and matching rules. A URL from a different origin is unclassified because the pasted file does not govern that origin. Invalid URLs remain visible as unclassified rows rather than disappearing from the batch.

Groups and precedence

This tool uses strict RFC 9309-style product tokens: exact matching, ignoring case. Matching specific groups are merged. If none match, wildcard groups apply; specific and wildcard groups are not combined. Consecutive user-agent lines share a group until an allow/disallow section begins. Blank lines, comments, sitemap and unsupported fields do not split a group.

Paths are case-sensitive. Matching starts at the beginning of the path and includes the query string, without the fragment. The longest normalized encoded pattern wins, counting wildcard and end-anchor symbols; Allow wins an equal-priority conflict. The exact root /robots.txt URI is implicitly allowed.

Encoding and wildcard behavior

An asterisk matches any sequence; a final dollar sign requires the URL path and query to end there. UTF-8 rule text is percent-encoded, hex digits are normalized, and percent-encoded unreserved ASCII is decoded for comparison. Encoded reserved slashes stay distinct from directory separators. Use %2A and %24 when those symbols are literal.

Invalid and unsupported lines are reported. Empty allow/disallow values do not block a path. Crawl-delay and nonstandard indexing rules are not enforced. This strict parser does not reproduce every search engine’s tolerance for malformed user-agent values or misspelled directives. It is not a drop-in copy of Google’s parser.

What the verdict can establish

An allow verdict means these pasted rules permit the URL; it does not mean the page is indexed. Robots rules are crawl preferences, not access controls. Network status codes, redirects, cached robots policies, authentication and live deployment are outside this offline test.

Limits: 500 KiB UTF-8 rules, 10,000 lines, 5,000 valid rules, 500 target URLs and 8,192 encoded characters per URL. Work stops at a bounded matching budget rather than guessing after excessive complexity. Each URL reports up to 100 matched rules while the verdict still considers all applicable rules; diagnostics show the first 200 lines with a total count. CSV downloads use CRLF and guard formula-like cells.

Questions & answers

Other Languages

References

• RFC 9309 - Robots Exclusion Protocol
• How Google Interprets the robots.txt Specification | Google Crawling Infrastructure  |  Crawling infrastructure  |  Google for Developers