⌘Ctrl K

robots.txt Tester

Check which URLs a robots.txt lets each crawler visit, and spot its mistakes

Uses our server
The file is fetched by this server; only public addresses on ports 80 and 443 are allowed. Or paste a robots.txt below.
robots.txt
Test URLs
Common crawlers
Analysis
How crawlers read robots.txt
One group per crawler
A crawler follows the groups naming its token (case-insensitive) and ignores the others, * included. It falls back to * only when no group names it.
Longest match wins
Among the rules of that group, the longest matching path decides; on a tie, Allow wins. Rules are prefixes: /admin also matches /admin-panel.
* and $
* matches any sequence of characters, $ at the end anchors the pattern to the end of the URL. Paths are case-sensitive and include the query string.
Missing file
A 4xx answer means no restrictions; a 5xx or no answer means the whole site is disallowed until the file is reachable again.
Not access control
robots.txt is public and only well-behaved crawlers obey it. To keep a page out of search results, use noindex on a page crawlers can fetch.