Crawling, Rendering, and Indexing

Discovery is knowledge of a URL

Discovery means a system becomes aware of a URL. Google describes finding pages through links on already known pages and through sitemaps, among other sources. Knowing that an address exists does not mean its content has been downloaded. A system may hold many known URLs while processing only some of them during a particular period. Google Search Central — In-depth guide to how Google Search works

A sitemap is a file that supplies information about pages and other files and their relationships. Google treats sitemap submission as a hint, not a guarantee that every listed URL will be crawled or indexed. A sitemap also does not override an access restriction. It describes possible resources; it is not a command that forces their inclusion in search results. Sitemaps overview

Crawling retrieves responses and resources

During crawling, automated software requests resources from servers. A response can contain HTML, an image, another supported file, a redirect, or an error. Google describes crawling as downloading text, images, and videos from discovered pages. Crawlers can revisit known URLs to detect changes, but discovery and revisit schedules are not identical for every page. Google Search Central — In-depth guide to how Google Search works

Retrieval is constrained by what the server returns and by applicable crawler rules. An HTTP response containing an ordinary page is different from a login screen or an error response at the same address. The address alone does not establish what was retrieved. Similarly, a server log entry showing a request is evidence of that request, not proof that the returned content was retained in an index.

Rendering interprets HTML and JavaScript

Rendering processes a document and its resources into a representation of the page. JavaScript can add text, links, or other content that is absent from the initial HTML response. Google’s JavaScript documentation distinguishes crawling, rendering, and indexing, and describes a rendering queue after the initial response is processed. This is Google-specific documentation, not a statement that every bot executes JavaScript. JavaScript basics

Google uses an evergreen Chromium-based rendering environment, but documents differences from a normal user’s browser session. Blocked resources cannot be fetched, and rendering does not reliably reproduce interactions such as clicking or scrolling to reveal content. A page can therefore appear differently in an interactive browser and in the rendered material available to Google. Rendering establishes an input for subsequent processing, not an indexing guarantee. JavaScript basics

Google also separates the HTML received from a server from the rendered HTML produced after processing scripts. Links can be extracted during initial processing and again after rendering. This explains why inspecting only the original response and inspecting a rendered document can reveal different content. Neither inspection alone establishes that Google ultimately selected the page for its index. JavaScript basics

Indexing analyzes and retains information

Indexing involves analyzing a page’s content and storing information in a search index. Google describes processing text, images, and video, and assessing signals that help it understand the page. The index is not simply a list of every downloaded URL. A crawled page may be excluded, and several accessible URLs may be treated as versions of the same underlying content. Google Search Central — In-depth guide to how Google Search works

Canonicalization identifies a representative URL among duplicate or substantially similar pages. Google documents redirects, rel="canonical", and sitemap inclusion as signals with different strengths; a declared canonical is not an absolute instruction. Canonicalization differs from prohibiting crawling or applying noindex. It concerns representation among alternatives rather than whether a crawler can retrieve a resource. Canonical URLs

Robots.txt controls crawler access

A robots.txt file supplies rules about which URLs compliant crawlers may request. Google explains that its primary purpose is managing crawler traffic, not preventing a page from appearing in search. A disallowed URL can still be discovered through external links and can sometimes appear without a content snippet. Blocking retrieval does not necessarily erase knowledge of the address. Robots.txt introduction

Robots.txt is also not an access-security mechanism. It is publicly readable, and some crawlers may ignore it. Private content protected by authentication is a different case from publicly accessible content covered by crawler preferences. Rules apply within a defined URL scope, so a file on one host does not automatically govern unrelated hosts. These limitations belong to crawler access, not to the meaning of the content. Robots.txt introduction

Noindex is an indexing directive

Google supports noindex in a robots meta tag or an X-Robots-Tag HTTP response header. The directive tells Google not to show the resource in search results after the directive is processed. A response header can apply to non-HTML resources as well as HTML. Google must be able to crawl the resource to discover its noindex instruction. Blocking indexing

Combining a robots.txt block with a noindex tag can prevent Google from seeing the tag. Google does not support noindex rules inside robots.txt. Its JavaScript documentation also cautions that an initial noindex can cause rendering to be skipped, so removing the directive through JavaScript is not a dependable reversal. The distinction is between access to the response and instructions discovered within it. JavaScript basics

Stage evidence and documented limits

The stages describe processing functions rather than a promise that every URL advances through a fixed timetable. A discovery record, successful fetch, rendered document, and indexed entry answer different factual questions. Serving results is another operation: Google selects results for a particular query after indexing. Inclusion in the index does not guarantee that a page appears for every relevant-looking search. Google Search Central — In-depth guide to how Google Search works

The distinction also limits what other website features establish. A navigation link can expose a URL, and structured data can describe its content, but neither proves indexing. The information architecture reference defines navigation relationships; the structured data reference defines machine-readable descriptions. The separate search visibility service page concerns a commercial offering. Google’s live documentation is a changing platform reference, not a universal specification for all search engines.

Frequently asked questions