Robots.txt Generator
Generate a valid robots.txt with allow/disallow rules, sitemap and crawl-delay. Copy and paste in seconds.
Tell search engines what to crawl. Build a clean robots.txt with sensible defaults and your sitemap.
Paths to block (one per line):
Run this tool
Free daily runs · Pro is unlimited.
How to use Robots.txt Generator
- Choose allow-all or block specific paths.
- Add your sitemap URL.
- Copy the robots.txt to your site root.
Frequently asked questions
Where does robots.txt go?
At the root of your domain, e.g. https://example.com/robots.txt
Does it prevent indexing?
robots.txt controls crawling, not indexing. Use a noindex meta tag to keep a page out of results.
Is it free?
Yes, free with no signup.
Automate this at scale
MarsGeo Pro adds bulk audit, programmatic SEO page generation and API access.
See plansRelated tools
XML Sitemap Generator
Paste your URLs and generate a valid XML sitemap ready to submit to Google Search Console.…
Open →Canonical Tag Checker
Paste a page's HTML and instantly see its canonical tag, title, meta description, H1 and O…
Open →Meta Description Generator
Turn a paragraph into a clean, length-checked meta description with a live SERP preview.…
Open →SEO Title Generator
Generate title tag variants for a keyword and see how they look in Google with a pixel-wid…
Open →Open Graph Generator
Generate Open Graph and Twitter Card meta tags so your link previews look right on social …
Open →Link Extractor
Paste HTML and extract every link, see internal vs external, and spot the ones you should …
Open →