ToolBirch

Robots.txt Generator

Build a robots.txt from your crawl rules.Web & SEO Tools

Robots.txt Generator

Build a robots.txt from your crawl policy. Format: one line per rule, e.g. User-agent: * / Disallow: /private / Allow: /public / Sitemap: https://…

Example: User-agent: * / Disallow: /admin / Allow: /public/ / Sitemap: https://example.com/sitemap.xml

About this tool

User-agent, Disallow, Allow and Sitemap lines become a correctly-ordered robots.txt, with the sitemap declarations grouped at the end as the convention expects.

How to use the Robots.txt Generator tool

  1. List your rules. One per line: User-agent, Disallow, Allow.
  2. Generate. A default wildcard User-agent is added if you omit one.
  3. Upload it. Place the output at the root of your domain.

Supported formats

Input

  • Plain text or file input

Output

  • Converted or analysed output

Inputs are validated before anything is processed. Invalid values and unsupported files are rejected with a clear error instead of failing silently.

Frequently asked questions

Does robots.txt block crawlers?

It instructs well-behaved crawlers; it is not access control. Protect private areas with authentication.

Where does the file go?

At the root of your domain, always.

How do sitemap lines work?

They are collected and emitted at the end, one per line.

Privacy

Everything you enter stays in this browser tab. There is no upload endpoint for tool inputs, no account, and no server-side copy of your text or files. Results are produced locally and disappear when you close the page.

No account is required and no file is ever uploaded. See the privacy policy for details about analytics and the limited data this site stores.

Known limitations

  • Crawl-delay directives are passed through unvalidated.

Related tools

Other free tools from the same collection.

Meta Tag GeneratorGenerate complete head meta markup from your page values.Open Graph GeneratorGenerate og: and twitter: card markup for rich previews.Sitemap ValidatorValidate sitemap XML structure, URLs and dates locally.Schema Markup GeneratorGenerate JSON-LD schema for common page types.