Crawling and indexing: robots.txt, sitemaps and HTTP status

Google shows a page only after three stages: Googlebot downloads the URL (crawling), Google processes the content and decides whether to store it in the index (indexing), and on each search it picks which indexed pages to show. When a page stays out of the index, the causes on the site's side are few and all of them can be checked: robots.txt blocks crawling, the page has noindex, the canonical URL points to another page, the server doesn't return 200, the content looks like an error page (soft 404), the content only appears after JavaScript runs, or Google crawled the page and chose not to index it. The Page indexing report in Search Console tells you which of these reasons applies to each URL.

Scope: this guide covers Google Search and is based on the Google Search Central documentation and the Search Console help pages, consulted in October 2026. Bing and other search engines follow similar standards (robots.txt, for example, is standardized in RFC 9309), but the behavior described here is Google's.

Crawling, indexing and serving

Googlebot is the Google program that downloads pages. It discovers URLs by following links from pages it already knows and by reading sitemaps. During crawling, Google renders the page and runs the JavaScript with a recent version of Chrome (Google Search Central, how Search works).

During indexing, Google analyzes text, images and videos and stores the result in the index. Pages with the same or similar content form a group, and Google picks one of them as the canonical: the version that can appear in results.

When serving results, Google picks the indexed pages it considers most relevant to the search. The same documentation says Google doesn't guarantee it will crawl, index or serve a page, and that it doesn't accept payment to crawl more often or rank higher.

This split helps with diagnosis. A crawling problem stops Google from reading the page; an indexing problem happens after Google has read it. The controls are different too: robots.txt acts on crawling, while noindex and canonicals act on indexing. Structured data works at the serving stage, making the page eligible for rich results such as breadcrumbs and prices, and it only counts on pages Google can crawl and index. The structured data guidelines ask that these pages not be blocked by robots.txt, noindex or any other access control. The article on structured data covers that part.

robots.txt controls crawling, not indexing

robots.txt is a text file at the root of the site (https://www.example.com/robots.txt) that says which paths each crawler may crawl. The rules apply only to the protocol, host and port where the file is served (Google, how Google interprets robots.txt).

User-agent: *
Disallow: /cart/
Disallow: /search

Sitemap: https://www.example.com/sitemap.xml

Google describes the file as a way to manage crawler traffic, so the server isn't overloaded and unimportant or duplicate pages aren't crawled. The same page warns that it is not a way to keep a page out of Google: a blocked URL can still be indexed if other pages link to it, but it shows up without a description (Google, introduction to robots.txt). In Search Console, this case shows up as "Indexed, though blocked by robots.txt".

To remove a page from the index, Google lists three options: noindex, password protection on the server, or removing the page. A noindex rule written inside robots.txt is not supported by Google (Google, block indexing with noindex).

An error in robots.txt itself can stop crawling of the whole site. If the file returns a 5xx error, Google stops crawling the site for 12 hours while it keeps trying to fetch it. After that, for up to 30 days, it uses the last good version it has cached. The file also has a 500 KiB limit; anything beyond that is ignored. Google usually caches robots.txt for up to 24 hours, so a fix can take a day to take effect.

Allowing or blocking AI crawlers such as GPTBot and ClaudeBot also goes through robots.txt. The article on AI crawlers in robots.txt covers the name of each crawler and the difference between search, user-triggered and training crawlers.

noindex: meta robots and X-Robots-Tag

noindex is the rule that asks Google not to show the page in results. It can go in a meta tag in the HTML or in the X-Robots-Tag HTTP header, with the same effect (Google, block indexing with noindex).

<meta name="robots" content="noindex">

X-Robots-Tag: noindex

The header is how you apply the rule to non-HTML files such as PDFs, images and videos. In the meta tag, name="robots" applies to every search engine that supports the rule, and name="googlebot" applies only to Google.

The trap is combining it with robots.txt. For noindex to work, Google has to crawl the page and read the rule. If robots.txt blocks the URL, Googlebot never sees the noindex, and the page can stay indexed through links from other sites (Google, robots meta tag and X-Robots-Tag specifications). If you want a page out of the index, leave crawling open until Google has read the noindex.

Besides noindex, Google supports nofollow (don't follow the page's links), none (same as noindex and nofollow together), nosnippet, max-snippet, max-image-preview, max-video-preview, unavailable_after, indexifembedded, notranslate and noimageindex. When two rules conflict, the more restrictive one wins. noarchive and nocache are ignored.

On JavaScript sites, noindex needs extra care. If the original HTML has noindex, Google may skip rendering, and a script that removes the rule later may have no effect. If you want the page indexed, the noindex can't be in the original code (Google, JavaScript SEO basics).

Any change to these rules takes effect only after Googlebot crawls the page again. For less important pages, the documentation says this can take months; URL Inspection lets you request a recrawl.

Canonicals: which URL in a group of duplicates appears

Duplicates are different URLs with the same or nearly the same content. They come from http and https, with and without www, with and without a trailing slash, from campaign parameters (?utm_source=) and from online store filters. Google groups these URLs and picks one as the canonical. You state your preference; the final choice is Google's.

<link rel="canonical" href="https://www.example.com/product/blue-sneakers">

Signals add up: a canonical tag and a sitemap pointing to the same URL reinforce the choice. The problem is a contradictory signal, such as a sitemap that lists one URL and a canonical tag that points to another. Google also advises against using robots.txt or noindex to choose a canonical: the first lets the URL be indexed without content, and the second removes the page from search entirely (Google, how to specify a canonical URL).

URL Inspection in Search Console shows the canonical you declared and the one Google selected. When they differ, the Page indexing report shows the reason "Duplicate, Google chose different canonical than user" (Search Console help, URL Inspection).

When the duplicate should go away, as in a move from http to https, the right signal is a redirect. The article on redirects, chains and loops covers the redirect types and how to test them.

Sitemap: the list of URLs you want indexed

A sitemap is a file, usually XML, that lists the site's URLs. Google treats a submission as a hint: it may not even download the file and doesn't promise to crawl everything in it (Google, build and submit a sitemap). The rules in the documentation:

• Each file holds up to 50,000 URLs or 50 MB uncompressed. Beyond that, split it into several sitemaps and list them all in a sitemap index file.

• The encoding must be UTF-8.

• URLs must be absolute (https://www.example.com/page), because Google crawls exactly what is written.

• Only canonical versions go in, the ones you want in results.

• Google uses lastmod only when it is consistently and verifiably accurate. The date should reflect the page's last significant change. A sitemap that stamps every URL with today's date each time it is generated doesn't meet that bar.

• Google ignores priority and changefreq.

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://www.example.com/product/blue-sneakers</loc>
    <lastmod>2026-10-20</lastmod>
  </url>
</urlset>

There are three ways to submit it: the Sitemaps report in Search Console, the Search Console API, or a Sitemap: line in robots.txt.

The sitemap has nothing to do with llms.txt, a file proposed for language models. In June 2026 Google added to its documentation that llms.txt files are not needed for Google Search and don't affect visibility or ranking (Google Search Central, documentation updates). The article on llms.txt explains what it's for outside Google.

HTTP status: what Googlebot does with each response

The status code is the three-digit number the server returns along with the page. What each code means is defined in RFC 9110; how Google handles them is in Google's crawling documentation.

A soft 404 is a page that returns 200 but shows an error message or comes back empty, such as a "product not found" served with a success status. For the server everything is fine; for Google, it's an error page. The fix is to return 404 or 410 when the content doesn't exist, or to add real content if the page should exist. On large sites, the crawl budget guide points out that soft 404 pages keep getting crawled and use up that budget.

JavaScript: when content depends on rendering

On sites that build the page in the browser, the HTML the server delivers can be nearly empty. Google processes these pages in three phases: crawl, render and index. Every page with a 200 status goes into a rendering queue unless it has noindex. It may wait a few seconds or longer until a headless Chromium runs the JavaScript. Pages with a status other than 200 may not be rendered at all (Google, JavaScript SEO basics).

Points the documentation calls out:

• Google only discovers links that are <a> elements with an href attribute. Navigation done only through a click event doesn't count as a link.

• Fragment routes (#/products) are not resolved reliably. Use the History API with real paths.

• In single-page applications, a route that doesn't exist usually returns 200. Google suggests redirecting through JavaScript to a URL that returns 404, or adding noindex to error screens.

• A noindex in the original HTML can make Google skip rendering.

Google still recommends server-side rendering or prerendering, because not every crawler runs JavaScript.

The reasons in the Page indexing report

In Search Console, the Page indexing report lists the URLs Google knows about and, for those not indexed, the reason. The names below are the ones in the English interface (Search Console help, Page indexing report).

Among indexed pages, the report can show two warnings: "Indexed, though blocked by robots.txt" and "Page indexed without content", when Google couldn't read the content.

Checklist: the page wasn't indexed

1. Open the URL in Search Console's URL Inspection. It shows the status, the last crawl and the canonical Google chose. The default result is the indexed version; the live test fetches the page on the spot.

2. Check the HTTP status. The URL has to return 200 on its own, without a redirect:

   curl -sI https://www.example.com/page

3. Check that robots.txt allows the path for Googlebot.

4. Look for noindex in the original HTML and in the X-Robots-Tag header. The command in step 2 shows the headers.

5. Check the canonical. The tag should point to the URL itself, or to the version you want indexed, and the sitemap should list that same URL.

6. Look at the HTML Google received (URL Inspection shows the HTML and the headers). The main content has to be there, not only appear after clicks in the browser.

7. Rule out a soft 404: the page has its own content and doesn't look like an error message.

8. Make sure the page can be found: it's in the sitemap and gets <a href> links from other pages on the site.

9. Read the reason in the Page indexing report and follow the matching row of the table above.

10. If the reason is "Crawled - currently not indexed", there is no technical block to remove. The recommendation here is to review the content, not the configuration: pages that are very similar to others on the site or have little text of their own are the first candidates. Google doesn't publish the exact criteria for this decision.

11. If the reason is "Discovered - currently not indexed", look at the server: 5xx and 429 responses make Google slow its crawl rate. The crawl budget guide was written for large sites (around 1 million pages, or 10,000 with daily changes, figures Google itself calls rough) and for sites with many URLs in this state.

12. After fixing, request indexing in URL Inspection. There is a daily request limit and a request doesn't guarantee indexing; for many URLs, use the sitemap.

How to check your site

Some of these checks can be done with WebVyze's free tools. The robots.txt validator checks the syntax, the declared sitemap and which crawlers (Google, Bing and AI crawlers, with search and training kept apart) may crawl a path. There's also the sitemap validator, the redirect checker and the structured data validator. These tools read the site's public pages; the exact reason a URL isn't indexed only shows up in Search Console.