Robots.txt Tester
Validate robots.txt rules
Source Input
Processed Output
What is the Robots.txt Tester?
One wrong line in robots.txt can quietly deindex your entire site. This tester fetches any domain's robots.txt (or parses pasted rules), validates the syntax line by line, warns about classic mistakes — blocking CSS/JS, conflicting directives, missing sitemap — and shows exactly which URL patterns each rule matches.
Frequently Asked Questions (FAQ)
Q: Does a Disallow rule remove a page from Google?
No. Disallow only prevents crawling — the URL can still be indexed from links. To deindex a page, use a noindex meta robots tag instead.
Q: How long until robots.txt changes take effect?
Google caches robots.txt for up to 24 hours. Changes typically apply on the next crawl after the cache expires.
Key Features
- Fetches any domain's live robots.txt
- Line-by-line syntax validation
- Blocks entire-site Disallow detection
- JavaScript/CSS blocking warnings
- Sitemap directive verification
How to Use the Robots.txt Tester
- 1Paste your input into the editor above — or start from the preloaded example.
- 2Adjust any options, then click Run to process instantly in your browser.
- 3Review the output panel, then copy, download, or save it as a reusable snippet.