robots.txt is a plain text file at the root of your domain that tells crawlers which parts of your site they may fetch. It is a convention, not a security mechanism, but every major search engine and most AI crawlers respect it.
Where it lives
The file must be at https://yourdomain.com/robots.txt. A file in a subfolder is ignored. You can only have one per origin.
The syntax, line by line
A robots.txt is made of groups. Each group starts with one or more User-agent lines followed by rules.
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /search
Sitemap: https://yourdomain.com/sitemap.xmlUser-agent: *applies to all crawlersDisallowblocks fetching;Allowcarves out exceptions and wins when it is more specificSitemappoints to your XML sitemap (not technically a robots rule, but widely supported)- Blank lines separate groups; comments start with
#
Wildcards and precedence
Most major crawlers support:
*matches any sequence (Disallow: /*?sort=)$anchors the end (Disallow: /*.pdf$)
When rules conflict, the most specific match wins — a longer path beats a shorter one. If a URL matches both an Allow and a Disallow of equal length, Allow wins.
Disallow is not noindex
This is the most common mistake. Disallow stops a crawler fetching a URL. If a page is already indexed, blocking it in robots.txt will not remove it — and Google cannot see a noindex tag on a page it is not allowed to fetch. To remove a page from the index, either return noindex (and allow crawling so it can be seen), or remove the page and return 404/410.
What to block
- Internal search results (
/search?q=) — they create infinite, low-value pages - Cart, checkout and account pages on e-commerce sites
- Filter and sort parameter combinations that explode into duplicates
- Staging or admin paths
- Session IDs and tracking-only paths when they cannot be canonicalised
What not to block
- CSS and JavaScript — if you block them, search engines cannot render your page correctly
- Your sitemap, or any page you want indexed
- Pages with a
noindextag you actually want removed - Whole sections "to be safe" — you lose crawl signals and usually create more problems
Test it before shipping
1. Fetch /robots.txt in a browser and confirm it returns 200 with text/plain.
2. Paste a URL into Google Search Console's robots.txt report to see which rule wins.
3. Confirm your sitemap URL is reachable and not itself blocked.
4. Check that CSS/JS paths are not disallowed.
Worked example
A small SaaS site wants to keep admin, api and search results out of the index but keep marketing pages and the browser rendering intact:
User-agent: *
Allow: /
Disallow: /admin
Disallow: /api
Disallow: /search
Sitemap: https://example.com/sitemap.xmlThat is deliberately minimal. Resist the urge to block /assets or /static — that breaks rendering.
Build one in seconds
Use the free robots.txt generator to assemble allow/disallow rules and a sitemap line, then copy it to your domain root. If you run WordPress, see the WordPress version.