Advertisement
← All Tools
Free · no sign-up

robots.txt Generator

Build a robots.txt with per-user-agent allow/disallow rules, crawl-delay and sitemap lines.

  1. Add rules per user-agent
  2. Add your sitemap URL
  3. Copy the robots.txt
Group 1
User-agent: *
Disallow:

Sitemap: https://example.com/sitemap.xml

Frequently asked questions

Where do I put the robots.txt file once generated?
Save it as robots.txt and upload it to the root of your domain (e.g. example.com/robots.txt) — search engines only look for it there.
Can I set different rules for different crawlers?
Yes — add a separate user-agent block for each crawler (like Googlebot or Bingbot) with its own allow and disallow rules, plus a catch-all block for everyone else.
Does robots.txt stop pages from appearing in search results?
Not reliably — it only tells crawlers not to fetch a URL. A disallowed page can still be indexed if it’s linked elsewhere; use a noindex meta tag to actually keep a page out of search results.
Should I include a sitemap line?
Yes — adding your sitemap.xml URL in robots.txt helps search engines discover it. Use the Sitemap Generator to build one if you don’t have it yet.
Do I need to install anything or sign up?
No. It runs entirely in your browser — no software install, no account, no email required.
Advertisement
Advertisement