Robots.txt Builder

Build a clean robots.txt file with crawl rules, sitemap lines, and optional AI crawler blocking without hand-writing syntax from scratch.

Site URL

Crawl delay (optional)

Allow paths (one per line)

Disallow paths (one per line)

Sitemaps (one per line)

Host (optional)

Auto-add `/sitemap.xml` from the site URL when the sitemap box is empty. Add `Disallow: /` rules for common AI crawlers (GPTBot, ClaudeBot, Claude-SearchBot, PerplexityBot, CCBot, Google-Extended, ChatGPT-User).

Preview updates live as you type. Rules are generated locally in your browser.

Local only Open site

Rule lines

Blocked paths

Sitemaps

Robots.txt Builder

Build a clean robots.txt file with crawl rules, sitemap lines, and optional AI crawler blocking without hand-writing syntax from scratch.

Create a starter robots.txt for a new site in a few clicks.

Block specific paths or crawler families without formatting mistakes.

Add sitemap and crawl-delay lines while keeping the file readable.

Typical workflow Set your rules, review the generated text, then copy the finished file into your web root.