Robots.txt tester
Paste your robots.txt file and the URLs to check: the tool applies the rules like Googlebot (most specific group, wildcards, longest rule) and tells you what is blocked, rule by rule. Everything runs in your browser.
| URL | Verdict | Rule applied |
|---|
What a robots.txt tester does
A robots.txt tester checks whether a given URL is allowed or blocked for a crawler, before a mistake costs you traffic. Googlebot first picks its rule group: the one whose User-agent line matches its name most precisely, otherwise the star group. It never uses two groups at once. Inside the group, the longest rule wins, and at equal length, Allow beats Disallow. The star stands for any sequence of characters and the dollar sign marks the end of the URL.
Classic pitfalls
A Disallow does not deindex: it prevents crawling, but a blocked page can stay indexed if it receives links. To deindex, you need a noindex and the page must stay crawlable. Another trap: Disallow: / blocks the whole site, while an empty Disallow forbids nothing. Finally, Crawl-delay is ignored by Google.
What this tool tests
The tool reproduces the matching rules documented by Google: group selection, wildcards, priority to the longest rule. Other search engines may interpret some edge cases differently, but Bing now follows very similar logic.