Where do I put the generated file?

Save it as robots.txt at the root of your domain (e.g. https://example.com/robots.txt) — it must not live in a subfolder.

Does robots.txt actually block a page from being indexed?

Not reliably — Disallow only asks crawlers not to fetch a page; a blocked page can still get indexed if other sites link to it. Use a noindex meta tag on the page itself to guarantee it stays out of search results.

Robots.txt Generator

Generate a robots.txt file with custom Allow/Disallow rules per user-agent and a sitemap link.

Top ad slot

User-agent

Disallow (one path per line) Allow (one path per line, optional)

Sitemap URL (optional)

robots.txt

User-agent: *

Related tools

Meta Title Length Checker
Meta Description Length Checker
Slug Generator
UTM Builder

Bottom ad slot

How it works

1
Set a User-agent

Use * for all crawlers, or a specific bot name.
2
List Disallow and Allow paths

One path per line — add more user-agent blocks if needed.
3
Add a sitemap URL (optional)

The file builds live as you fill in the fields.

FAQ

Where do I put the generated file?: Save it as robots.txt at the root of your domain (e.g. https://example.com/robots.txt) — it must not live in a subfolder.
Does robots.txt actually block a page from being indexed?: Not reliably — Disallow only asks crawlers not to fetch a page; a blocked page can still get indexed if other sites link to it. Use a noindex meta tag on the page itself to guarantee it stays out of search results.