Robots.txt tester
Fetch and validate robots.txt and test whether a URL is allowed.
This robots.txt tester fetches the file from your domain, validates its syntax, and lets you test a specific URL against the rules to see whether crawlers are allowed or blocked. It interprets User-agent, Allow, and Disallow directives the way search engines do, so you can confirm important pages are crawlable and sensitive paths are not. A stray Disallow: / can quietly remove your whole site from search, which makes verification worth doing after any change. To guard against that drifting over time, SJ Monitor's HTTP monitoring can watch the file and alert you to changes.
Frequently asked questions
What does Disallow: / do?
It blocks compliant crawlers from the entire site, which can deindex you from search engines. Always confirm this is intentional before deploying.
Why is a page I want indexed being blocked?
A broader Disallow rule may be matching it, or a more specific Allow is missing. The tester shows which rule applies to the URL you check.
How do I catch accidental robots.txt changes?
Deploys can overwrite the file without anyone noticing. SJ Monitor's continuous HTTP monitoring detects changes to robots.txt and alerts you so blocks do not slip through.
Does robots.txt keep a page out of Google?
Not reliably. Disallow stops crawling, but a blocked URL can still be indexed if other sites link to it. To keep a page out of results, allow crawling and use a noindex meta tag or header instead.
Where should robots.txt live?
At the root of each host, exactly at /robots.txt — crawlers won't look anywhere else, and rules don't apply across subdomains. blog.example.com needs its own file separate from example.com.
Should I list my sitemap in robots.txt?
Yes. A Sitemap: line pointing to your XML sitemap helps crawlers discover all your URLs, and it's independent of the Allow/Disallow rules.
More Website & HTTP tools tools
Want this checked automatically and around the clock? Create a free SJ Monitor account and we'll alert you the moment something changes.