Robots.txt Checker
Retrieve robots.txt, list user-agent groups, disallow rules and sitemap references.
Enter a value above and run the check to see results here.
How it works
Retrieve robots.txt, list user-agent groups, disallow rules and sitemap references.
Frequently asked questions
Is the Robots.txt Checker tool free to use?+
Yes. The Robots.txt Checker runs directly in NetInsight with no signup required. Free-tier usage is rate limited to keep the service fast for everyone.
Are my queries logged?+
Diagnostic inputs are processed to produce your result and are not sold or shared. See our Privacy Policy for full detail on retention.
How accurate are the Robots.txt Checker results?+
Results are fetched live from authoritative sources each time you run a check, so they reflect the current state rather than a cached snapshot.