Loading...
Loading...
Generate a robots.txt file for your website instantly. Free online robots.txt generator to control search engine crawlers.
It tells well-behaved crawlers which paths on your site they may or may not request. It's a request, not an enforcement mechanism — it doesn't password-protect anything and doesn't remove pages already indexed by search engines.
The syntax is small but easy to get subtly wrong — a missing slash, a rule under the wrong user-agent block, or a sitemap URL in the wrong format can silently misconfigure crawling. A generator keeps each rule tied to the right user-agent automatically.
When you have staging or admin paths you don't want crawled, filtered/faceted URLs creating near-duplicate pages that waste crawl budget, or when you want one crawler (like an image bot) to follow different rules than the rest.
1) Add a rule for each user-agent you want to target 2) Specify allowed and disallowed paths for each rule 3) Add your sitemap URL 4) Download or copy the generated robots.txt content.
Confusing Disallow with removing a page from search results. Blocking a URL in robots.txt only stops crawling — if the page is already indexed, or if other pages link to it, it can still appear in search results without a snippet. Use a noindex meta tag if you actually want a page removed from search results.
Yes — a common mistake is accidentally disallowing a path that contains CSS or JavaScript files search engines need to render the page properly, which can hurt how the page is understood and ranked. Double-check broad Disallow rules like /assets/ or /static/ don't catch files you need crawled.
Adding a Sitemap: line points crawlers directly to your sitemap.xml, which helps them discover new and updated pages faster than relying on links alone.
Robots.txt controls whether a page gets crawled at all; Meta Tag Generator and Open Graph Generator control how a page is described once it's indexed or shared. All three are commonly set up together when launching a new page.