Robots.txt Generator
Generate robots.txt files for your website instantly. Control search engine crawlers, set crawl delays, block specific paths, and add sitemap URLs. Supports all major bots including Googlebot, Bingbot, and GPTBot. Free online robots.txt generator.
Frequently Asked Questions
What is a robots.txt file?
A robots.txt file is a text file placed at the root of your website (e.g., example.com/robots.txt) that tells search engine crawlers which pages or files they can or cannot request. It's part of the Robots Exclusion Protocol (REP) and is the first thing most crawlers check before visiting your site.
Does robots.txt block pages from being indexed?
No. robots.txt only prevents crawlers from visiting a page, not from indexing it. If a page is linked from other sites, Google may still index it (showing it in results without crawling the content). To truly prevent indexing, use the noindex meta tag or HTTP header instead.
What is the difference between Allow and Disallow?
Disallow tells crawlers not to access specific paths. Allow explicitly permits access and is useful for overriding a broader Disallow rule. For example, Disallow: /private/ blocks everything under /private/, but Allow: /public/ within a Disallow: / rule lets crawlers access /public/ while blocking everything else.
Should I block AI crawlers like GPTBot?
It depends on your goals. Blocking GPTBot, Bytespider, and other AI crawlers prevents your content from being used to train AI models. However, it may also reduce your visibility in AI-powered search results. Most sites allow Googlebot and Bingbot (for search indexing) but make individual decisions about AI training bots. Review each bot's documentation before deciding.
What is crawl-delay and should I use it?
Crawl-delay specifies the number of seconds a crawler should wait between requests. It's primarily respected by Bingbot and some other bots โ Googlebot does not support it. Use it if your server is overwhelmed by crawler traffic, but avoid setting it too high (e.g., 10+ seconds) as it significantly slows down indexing.
Where should I place the robots.txt file?
The file must be named robots.txt (lowercase) and placed at the root of your domain: https://example.com/robots.txt. It must be accessible via HTTP with a 200 OK status and text/plain content type. Subdirectories are not supported โ one file per domain.
Is this tool free?
Yes, completely free with no sign-up required. All generation happens in your browser โ nothing is sent to any server.