THE FEUTEX TOOLKIT

Online Robots.txt Generator

Generate a clean robots.txt with user-agent groups, allow/disallow rules, crawl-delay, host and sitemap lines. Runs fully offline in your browser.

Quick builder (optional)

Use these to insert common lines; you can still edit the main rules input below.

Rules input

Supported directives: User-agent, Disallow, Allow, Crawl-delay, Host, Sitemap. Lines starting with # are ignored.

robots.txt output

Output is normalized and compact. Sitemap lines are deduplicated and placed at the bottom.

Privacy: runs locally in your browser. No uploads, no tracking scripts.

How to use

Use the input box to describe your rules, then generate your final robots.txt.

Start each group with one or more User-agent: lines.
Add rules like Disallow:, Allow:, and Crawl-delay: under that group.
Add Sitemap: (and optional Host:) anywhere—this tool places them at the bottom and deduplicates sitemap lines.
Click Generate, then copy or download the output.

What does robots.txt do?

It tells crawlers which paths they may or may not fetch on your site; it doesn’t enforce security.

How do I allow everything?

Use User-agent: * and an empty Disallow: line.

How do I block the whole site from crawling?

Use User-agent: * and Disallow: /.

Can I add multiple sitemap URLs?

Yes—add multiple Sitemap: lines; this tool deduplicates identical URLs.

Do I need a separate group for Googlebot?

Only if you want different rules; otherwise User-agent: * applies to most crawlers.

Where should Sitemap lines go in robots.txt?

Anywhere is valid, but most sites place them at the bottom for readability.

Is the Host directive required?

No—Host: is not part of the official standard and is ignored by many bots.

Browse related tools by task. Use the improved core tools for a consistent workspace with file import, downloads and clear validation errors.