Question 1

How does robots.txt rule matching work?

Accepted Answer

Robots.txt rules are matched by specificity. When a crawler checks a URL, it finds all Allow and Disallow rules that match the URL path and applies the most specific one. If Allow and Disallow rules have equal specificity, Allow wins. For example, "Disallow: /products" and "Allow: /products/sale" — the URL /products/sale/item.html would be allowed because /products/sale is more specific. A single asterisk (*) matches any user-agent. Google also supports the $ wildcard to match end-of-string.

Question 2

Can I test my live robots.txt file directly?

Accepted Answer

Yes. Enter your website URL in the tester and it will fetch your live robots.txt file from the standard location (yourdomain.com/robots.txt). Alternatively, paste your robots.txt content directly to test changes before deploying them. Google Search Console also has a built-in robots.txt tester under the URL Inspection tool that shows you the exact rules Google is reading. Testing before deployment is important — a single typo like "Disallow: /" blocks your entire site from Googlebot.

Robots.txt Tester

What gets checked