← Blog · By Mars LLC editorial team · September 24, 2026 · Editorial standards

What is robots.txt (and how to write one)

robots.txt controls what crawlers may fetch. Learn the syntax, the common mistakes, and how it differs from noindex.

robots.txt is a plain text file at the root of your domain that tells crawlers which parts of your site they may fetch. It is a convention, not a security mechanism, but every major search engine and most AI crawlers respect it.

Where it lives

The file must be at https://yourdomain.com/robots.txt. A file in a subfolder is ignored. You can only have one per origin.

The syntax, line by line

A robots.txt is made of groups. Each group starts with one or more User-agent lines followed by rules.

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /search

Sitemap: https://yourdomain.com/sitemap.xml
  • User-agent: * applies to all crawlers
  • Disallow blocks fetching; Allow carves out exceptions and wins when it is more specific
  • Sitemap points to your XML sitemap (not technically a robots rule, but widely supported)
  • Blank lines separate groups; comments start with #

Wildcards and precedence

Most major crawlers support:

  • * matches any sequence (Disallow: /*?sort=)
  • $ anchors the end (Disallow: /*.pdf$)

When rules conflict, the most specific match wins — a longer path beats a shorter one. If a URL matches both an Allow and a Disallow of equal length, Allow wins.

Disallow is not noindex

This is the most common mistake. Disallow stops a crawler fetching a URL. If a page is already indexed, blocking it in robots.txt will not remove it — and Google cannot see a noindex tag on a page it is not allowed to fetch. To remove a page from the index, either return noindex (and allow crawling so it can be seen), or remove the page and return 404/410.

What to block

  • Internal search results (/search?q=) — they create infinite, low-value pages
  • Cart, checkout and account pages on e-commerce sites
  • Filter and sort parameter combinations that explode into duplicates
  • Staging or admin paths
  • Session IDs and tracking-only paths when they cannot be canonicalised

What not to block

  • CSS and JavaScript — if you block them, search engines cannot render your page correctly
  • Your sitemap, or any page you want indexed
  • Pages with a noindex tag you actually want removed
  • Whole sections "to be safe" — you lose crawl signals and usually create more problems

Test it before shipping

1. Fetch /robots.txt in a browser and confirm it returns 200 with text/plain.

2. Paste a URL into Google Search Console's robots.txt report to see which rule wins.

3. Confirm your sitemap URL is reachable and not itself blocked.

4. Check that CSS/JS paths are not disallowed.

Worked example

A small SaaS site wants to keep admin, api and search results out of the index but keep marketing pages and the browser rendering intact:

User-agent: *
Allow: /
Disallow: /admin
Disallow: /api
Disallow: /search

Sitemap: https://example.com/sitemap.xml

That is deliberately minimal. Resist the urge to block /assets or /static — that breaks rendering.

Build one in seconds

Use the free robots.txt generator to assemble allow/disallow rules and a sitemap line, then copy it to your domain root. If you run WordPress, see the WordPress version.

Try the free tool

Generate a valid robots.txt with allow/disallow rules, sitemap and crawl-delay. Copy and paste in seconds.

Open Robots.txt Generator