About the Robots.txt Tester
The Robots.txt Tester fetches a site's robots.txt, parses every user-agent group and lets you test a specific path against the rules — with the same matching logic Google uses: longest rule wins, Allow beats Disallow on ties, including * wildcards and $ end anchors.
It also lists all declared sitemaps (jump straight into the Sitemap Checker) and flags dangerous states like a 5xx robots.txt, which makes Google pause crawling of the whole site.
robots.txt controls crawling, not indexing. A disallowed URL can still appear in Google (without a snippet) if other pages link to it. To keep a page out of the index, use a meta robots noindex tag and let it be crawled.
Google applies the most specific (longest) matching rule; if an Allow and a Disallow match with equal length, Allow wins. This tester implements exactly that logic, including * wildcards and $ end-anchors.
Googlebot for Google Search, Googlebot-Image for image search, Bingbot for Bing. Testing GPTBot, ClaudeBot or CCBot shows whether AI crawlers are allowed to read your content.