robots.txt Tester

How the decision is made

Two rules decide everything, and both surprise people.

One group applies, not all of them. A crawler uses the group whose User-agent matches it most specifically, and ignores every other group — including the wildcard one. So a Disallow: / under User-agent: * has no effect on Googlebot if a User-agent: Googlebot group exists anywhere in the file.

Within that group, the longest matching rule wins. This is why Allow: /wp-admin/admin-ajax.php beats Disallow: /wp-admin/ — it is the longer match, so the more specific instruction. It is also the reason that pairing is in almost every WordPress robots.txt.

Crawling is not indexing

A blocked URL can still appear in search results. Blocking prevents fetching, not listing: if enough links point at a page, it can be indexed on the strength of those links alone, shown with no description because the crawler was never allowed to read it.

To keep a page out of the index you must let it be crawled and serve a noindex tag. Doing both — blocking in robots.txt and adding noindex — achieves the opposite of what is intended, because the crawler is never permitted to read the tag telling it to stay away.

What belongs in a WordPress robots.txt

Very little. WordPress serves a sensible virtual file already, and most hand-written additions cause more trouble than they prevent:

• Do not block /wp-content/uploads/ unless you want images out of image search.
• Do not block CSS or JavaScript. Google renders pages, and a page it cannot style is a page it judges on what it managed to load.
• Do declare the sitemap.

Related

If a page is missing from search, robots.txt is one of four possible causes and rarely the first — start with site not showing in Google. For what the file is and how WordPress generates it, see the SEO checklist.