Robots.txt Checker
Fetch and parse a live site’s robots.txt — user-agent rules and listed sitemaps.
Related tools
How it works
- 1
Enter a domain
Just the site, e.g. https://example.com.
- 2
robots.txt fetched from its root
The tool requests /robots.txt on that domain automatically.
- 3
Rules parsed and shown
Per user-agent Allow/Disallow rules, plus any Sitemap lines.
FAQ
- What if the site has no robots.txt?
- That’s reported directly — a missing robots.txt is valid (it means "no restrictions stated"), not an error.
- Does this tool store the domains I check?
- No — nothing is logged or stored.