Build a clean robots.txt file with crawl rules, sitemap lines, and optional AI crawler blocking without hand-writing syntax from scratch.
Preview updates live as you type. Rules are generated locally in your browser.
Robots.txt Builder assembles a valid robots.txt from allow and disallow paths, sitemap lines, an optional crawl delay and host directive, with presets for a starter file and for hiding admin paths.
Robots.txt controls crawling, not indexing, and the difference matters enormously. Disallowing a path tells well-behaved crawlers not to fetch it. It does not remove the URL from search results, and a disallowed page that other sites link to can still be indexed, listed with no description because the crawler was never allowed to read it.
To keep a page out of the index you need a noindex meta tag or header, and the crawler has to be allowed to fetch the page to see it. Disallowing a page you have also marked noindex is self-defeating, because the crawler never reads the noindex.
