ROBOTS.TXTFive mistakes to avoid
SELLWRIGHTA robots.txt file is a short text file that tells crawlers which URLs on your site they may request. It must be named robots.txt and sit at the top level of the host, such as https://www.example.com/robots.txt, and it only applies to that protocol, host and port, so each subdomain needs its own. It is easy to edit and easy to get wrong: one line can stop Google crawling an entire site. These are the five mistakes worth checking for.
1. Using Disallow to keep pages out of Google
Disallow stops crawling, not indexing. Google describes robots.txt as a way to manage crawler traffic, not a way to keep pages out of search. If other pages link to a blocked URL, Google can still index the address without visiting it, and the result appears without a description.
To keep a page out of search, use a noindex rule, either <meta name="robots" content="noindex"> in the head or an X-Robots-Tag: noindex header, or put the page behind a password. The catch is that Google has to crawl the page to see noindex, so a page that is both disallowed and tagged will never have its tag read. Allow crawling, add noindex and let Google recrawl. A noindex line inside robots.txt does nothing, because Google doesn't support it.
2. Leaving staging rules in place after launch
Staging sites are often blocked from crawling, and the block can travel to the live site at launch. Two lines are enough:
User-agent: *
Disallow: /Disallow: / matches the root and every URL below it, so every crawler is told to stay off the whole site. Google's site move guidance covers this: if you block crawling during development, prepare the robots.txt you will use at launch in advance. On launch day, check that:
- The live robots.txt has replaced the staging one, with no
Disallow: /for all user agents. - Development
noindexrules are gone from templates and response headers. - On WordPress, the "Discourage search engines from indexing this site" box is cleared. Since version 5.3 it adds a
noindexmeta tag rather than a robots.txt rule.
Protect staging with a password instead. Robots.txt is a request that reputable crawlers follow, not access control, and a password-protected staging site has no crawl rules to carry over at launch.
3. Blocking CSS and JavaScript
Don't block the files your pages need to render. Google says you can block unimportant images, scripts or styles, but if their absence makes a page harder for Google to understand, leave them crawlable, or Google won't analyse those pages well. Watch for rules that block a whole theme, plugin or assets folder.
To see what Google gets, run a live test in the URL Inspection tool in Search Console and open the tested page: it shows a screenshot of the rendered page and the page resources Google loaded.
4. Getting the case of a path wrong
Paths in robots.txt are case-sensitive, so Disallow: /Private/ doesn't block /private/. In Google's own example, a rule for /fish matches /fish.html and /fish/salmon.html, but not /Fish.asp. Field names such as Disallow can be written in any case; the values can't. Copy paths exactly as they appear in your URLs.
Two more syntax points: * matches any sequence of characters and $ marks the end of a URL. When an allow rule and a disallow rule both match, Google applies the most specific one, judged by the length of the path, and if they still conflict, the least restrictive.
5. Leaving out the sitemap line
Add a Sitemap: line with the full URL of your XML sitemap. It isn't required, but Google, Bing and other major search engines support it, it isn't tied to any user agent, and you can list several. Use the complete address, including https://. A simple file for a typical marketing site looks like this:
User-agent: *
Disallow: /search/
Sitemap: https://www.example.com/sitemap.xmlThe Disallow: /search/ line keeps crawlers out of internal search results, a typical set of unimportant pages. Leave it out if your site has no such pages.
Where to start
Open /robots.txt on your live domain and on every subdomain, and read each line. Then check the robots.txt report in Search Console, which shows the files Google found for the top 20 hosts on your site, when each was last crawled, and any warnings or errors. If anything looks wrong, our SEO and AI search service can review it with the rest of your crawl setup, or start with a website review.
FAQ
Frequently asked questions
Do we need a robots.txt file at all?
No. Without one, crawlers may request every URL, which suits many small sites. Add one when you want to keep crawlers away from unimportant areas, such as internal search results, or to point to your sitemap. Google treats a missing file that returns a 404 as having no crawl restrictions.
What happens if robots.txt returns a server error?
Crawling pauses. For the first 12 hours, Google stops crawling the site while it keeps trying to fetch the file, then uses the last good version it cached for up to 30 days. Most 4xx responses, by contrast, are treated as if no robots.txt existed, so crawling continues without restrictions.
Does robots.txt keep private pages secure?
No. Robots.txt is a request, not access control: Googlebot and other reputable crawlers follow it, but other crawlers may not. Google recommends password protection for anything you need to keep private. That applies to staging sites, client areas and internal documents alike.
Can we set a crawl delay for Google?
Not in robots.txt. Google doesn't support the crawl-delay field and ignores it. Its crawlers slow down on their own when a server returns 5xx or 429 errors, and Google advises against using 401 or 403 responses to limit crawling.
How quickly does a robots.txt change take effect?
Allow about a day. Google generally caches robots.txt for up to 24 hours, and longer if it can't refresh the file. In an emergency, such as an accidental Disallow: / on the live site, the robots.txt report in Search Console lets you request a recrawl of the file.
