Search Engine Indexing: How Google Stores Pages

SEO course › Part 1: SEO foundations

Search engine indexing is the process of analyzing a crawled web page and storing information about it in a huge database called the index. When someone searches, Google looks up matching pages in this index, not on the live web. If your page is not indexed, it cannot appear in search results at all.

Long aisle of library bookshelves — Indexing works like a library catalogue of every page a search engine has stored

This lesson explains what happens during indexing, how Google handles duplicate pages, why some pages are left out, and how to check and fix indexing problems. It is part of our free SEO course.

What is search engine indexing?

Think of the index as the catalog of a very large library. Crawling brings the books in the door. Indexing reads each one, notes what it is about, and files it so it can be found quickly.

Indexing comes after how search engines crawl websites and before how Google ranks pages. Google explains all three stages in its guide to how Google Search works.

What happens when Google indexes a page?

During indexing, Google processes and analyzes what is on the page. This includes:

The main text and the words used on the page.
Key elements such as the title tag and headings.
Images and videos, along with their alt text (a short text description of an image) and captions.
The language of the page and, for local businesses, signals about location.
Structured data, which is extra code that labels details such as prices, ratings or recipe cooking times.

Google also renders pages, which means it runs the JavaScript to see the page roughly as a visitor would. Content that only appears after rendering can still be indexed, but it may take longer.

Google completed its move to mobile-first indexing in 2023. This means it mainly uses the mobile version of your content for indexing and ranking.

How does Google handle duplicate pages?

Many sites have several URLs that show the same or very similar content. For example, example.com/page and example.com/page?ref=whatsapp may show identical content.

Google groups these duplicates together and picks one URL as the canonical, the main version it shows in search. The other versions are usually not shown.

You can suggest your preferred version with a canonical tag, a line of code in the page’s head section that points to the main URL. Google treats this as a strong hint, not a command, and may choose a different canonical if your other signals disagree.

Example: a food blogger’s recipe pages

Imagine a food blogger from Chennai who posts South Indian breakfast recipes. Each recipe has a normal page and a separate print-friendly page, and it also appears on several tag pages.

Google may see the recipe page and its print version as duplicates. If the blogger does nothing, Google chooses the canonical on its own, and it could pick the print version.

The blogger adds a canonical tag on each print page pointing to the main recipe URL, and links internally only to the main version. Now the signals agree, and Google is much more likely to index and show the right page.

Why are some pages not indexed?

Google does not promise to index every page it crawls. Common reasons include:

A noindex rule. A robots meta tag or HTTP header containing noindex tells search engines not to index the page.
Duplicate content. The page was treated as a duplicate and another URL was chosen as the canonical.
Thin or low-value content. Pages with very little useful content may be crawled but not indexed.
Errors. Pages that return 404 (not found), server errors or soft 404s (pages that say “not found” but return a normal status code) are left out.
Blocked by robots.txt. If Google cannot crawl a page, it cannot read its content.
Too new. Google may simply not have processed it yet.

How can you check whether a page is indexed?

Open Google Search Console and enter the page URL in the URL Inspection tool at the top.
Read the result. It tells you whether the URL is on Google and, if not, why.
If you have fixed a problem, click “Request indexing” to ask Google to crawl the page again.
Review the Page indexing report to see all indexed and non-indexed pages, grouped by reason.

Two statuses in that report confuse many beginners:

Status in Search Console	What it means	What you can do
Discovered – currently not indexed	Google knows the URL but has not crawled it yet	Add internal links to the page and be patient
Crawled – currently not indexed	Google crawled the page but chose not to index it for now	Make the content more useful and distinct, then wait

You can also search site:yourdomain.com in Google for a rough view of your indexed pages, but it is not a complete or exact count.

Common mistakes to avoid

Leaving a sitewide noindex setting on after moving a site from testing to live. In WordPress, this is the “Discourage search engines from indexing this site” checkbox.
Blocking a page in robots.txt while also adding noindex, so Google never sees the noindex rule.
Pointing canonical tags at pages that redirect or return errors.
Publishing many near-identical pages, such as city pages that only swap the town name.

Key takeaways

Search engine indexing means analyzing pages and storing them in the search index.
Only indexed pages can appear in search results, including in AI Overviews.
Google picks one canonical URL from each group of duplicate pages.
Noindex rules, duplicates, thin content and errors are common reasons pages are not indexed.
The URL Inspection tool and Page indexing report in Search Console show your indexing status.

Frequently asked questions

How long does it take Google to index a new page?

It can take anywhere from a few days to several weeks. Good internal links and a submitted sitemap help Google find new pages sooner.

Does indexing guarantee my page will rank?

No. Indexing only makes a page eligible to appear. Where it ranks depends on relevance, quality and many other factors.

Do pages need to be indexed to appear in AI Overviews?

Yes. Google has said a page must be indexed and eligible to be shown with a snippet to appear as a supporting link in AI Overviews. Read more about how AI Overviews affect traffic.

Previous lesson: How search engines crawl websites: a beginner’s guide

Next lesson: How Google ranks pages: ranking factors and algorithms explained