How Google Search Works: Crawling, Indexing and Ranking

Google Search works in three separate stages: crawling (finding pages), indexing (understanding and storing them), and serving (ranking them for a query). This guide is for founders, marketers, and developers who keep hearing "it's a crawling issue" or "it's a ranking problem" without knowing what those words actually mean. We walk through each stage the way Google's own engineers describe it, show where each one can break, and give you a diagnostic list to find the exact stage a page is stuck at. No invented ranking weightings, no secret factors, just the mechanics and where you have control.

The short version

• Crawled and indexed are not the same thing. A page can be crawled, understood, and still left out of the index. That gap is where most "why isn't my page ranking" problems actually live.
• The three stages run in order: crawling, then indexing, then serving. Diagnose them in that order too, because a serving problem you can't fix if the page was never indexed.
• Crawl budget is a real thing, but for most sites it is not the problem. It matters at scale, not for a 200-page site.
• robots.txt controls crawling, not indexing. Blocking a page there can keep its noindex tag from ever being seen.
• AI Overviews and AI search sit on top of the same index. If a page isn't indexed, it can't be crawled, understood, or cited by either search or AI.

What are the three stages of Google Search?

Crawling, indexing, and serving results. Google describes Search as three distinct stages, and a page has to clear each one in order before it can appear for a query.

Crawling is discovery: Googlebot fetches pages it has found. Indexing is comprehension and storage: Google renders the page, works out what it's about, and decides whether to keep it. Serving is retrieval and ranking: when someone searches, Google pulls matching pages from the index and orders them.

Keeping these separate is the single most useful habit in technical SEO. When a page has a problem, the first question is not "how do I rank it". It's "which stage is it stuck at." A page that was never crawled and a page that ranks on page three have completely different fixes, and confusing the two costs days.

Stage	What happens	Where you check it
Crawling	Googlebot discovers and fetches the URL	URL Inspection → "Crawled"; server log files
Indexing	Google renders, understands, and stores the page	URL Inspection → "Indexed"; Pages report
Serving	The page is retrieved and ranked for a query	Performance report; a live search

Google's overview of how Search works lays out the same three stages, and it's the reference we point clients to first. Our complete SEO guide puts these mechanics in the wider context of a growth strategy.

How does Google crawl the web?

By following links and reading sitemaps, then queueing URLs for Googlebot to fetch. Crawling starts with a list of URLs Google already knows, expands it every time it finds a link to somewhere new, and schedules fetches from that queue.

Discovery happens in a few ways. Links from pages Google already crawls are the main one. An XML sitemap.xml you submit in Search Console is another. So is a URL you enter manually in the URL Inspection tool.

Before fetching, Googlebot checks your robots.txt file. That file tells crawlers which paths they may request. If a path is disallowed there, Googlebot won't fetch it, which sounds like a clean way to keep a page out of Search, but it isn't, for a reason we'll get to.

Not every fetch succeeds. Server errors, timeouts, and 5xx responses all interrupt crawling, and a page that can't be fetched can't move to the next stage. If a page you care about isn't showing as crawled, start with server health and internal links pointing to it. Our technical SEO guide covers the crawl-path fixes in depth.

Does crawl budget actually matter for your site?

Usually not. Crawl budget, the number of URLs Googlebot will fetch on your site in a given window, is a genuine constraint, but it only becomes a bottleneck on large sites.

Google itself has said that most sites don't need to think about crawl budget. It becomes relevant when a site has a very large number of URLs, when pages change constantly and need frequent re-crawling, or when a slow server forces Googlebot to back off. Think large e-commerce catalogs, big news publishers, and sprawling programmatic sites.

When crawl budget is NOT your problem: if your site has a few hundred or a few thousand pages, a page missing from the index is almost never a budget issue. It's far more likely to be a quality, duplication, or discovery problem. Spending a week optimizing crawl efficiency on a 300-page site is effort aimed at the wrong stage.

The exception worth watching, even on smaller sites, is wasted crawling: faceted-navigation URLs, session parameters, and infinite calendar links can generate thousands of low-value URLs that soak up fetches. That's worth cleaning up because it's messy, not because you're about to hit a ceiling.

How does Google render a page before indexing it?

In two passes. Google first reads the raw HTML, then, for pages that need it, comes back later to run JavaScript and render the page the way a browser would.

This matters because a lot of modern sites build their main content with JavaScript — our guide to making JS sites crawlable and indexable covers how to confirm Googlebot sees the content anyway. When Googlebot first fetches the HTML, that content may not be there yet. The page goes into a queue for rendering, and only after rendering does Google see the fully built page. The raw HTML and the rendered HTML can differ, and Google indexes what it sees after rendering.

The practical risk: if your critical content, internal links, or canonical tags only appear after JavaScript runs, they depend on that second pass completing correctly. Server-side rendering or static generation removes that dependency by putting the content in the initial HTML. This is one of the more common failure points we see on React and single-page-application sites during audits — our guide to building search-friendly React applications covers the framework that solves it most often in practice.

Why do crawled pages not always get indexed?

Because crawling only earns a page the chance to be indexed, it doesn't guarantee it. Google renders and evaluates each crawled page and then decides whether to store it, and plenty of crawled pages don't make the cut.

The common reasons a crawled page stays out of the index:

• Duplicate or near-duplicate content. Google groups pages it sees as duplicates and usually keeps only one.
• A noindex tag. If the page serves a noindex directive, Google honors it and drops the page from the index.
• Thin or low-value content. Pages that add little may be crawled and then not selected.
• Canonicalization. If Google picks a different URL as the canonical version, the duplicate won't be indexed under its own address.
• Soft 404s. Pages that look empty or error-like get treated as not worth keeping.

This is why "Google crawled it" is never the end of the story. The Pages report in Search Console separates crawled-not-indexed from other states precisely so you can see this gap.

How does Google choose a canonical URL?

It picks one representative URL from a group of pages it considers the same or very similar. When several URLs show the same content, Google clusters them and selects a canonical, the version it indexes and shows in results.

Your rel="canonical" tag is a strong signal in that decision, but it's a hint, not a command. Google also weighs internal links, redirects, sitemap inclusion, and HTTPS. If your signals conflict, say your canonical points one way but your internal links and sitemap point another, Google may choose a canonical you didn't intend.

In our experience, most accidental de-indexing traces back to conflicting canonical signals rather than anything exotic. Make your canonical tag, internal links, and sitemap agree on one URL, and the guesswork mostly disappears.

How does Google serve and rank results?

It retrieves matching pages from the index and orders them using a set of ranking systems, all aimed at surfacing the most relevant, useful result for the query. Serving is the stage most people mean when they say "SEO," but it only operates on pages already in the index.

Google describes the signals conceptually rather than as a fixed formula, and so will we, anyone quoting exact percentage weightings is inventing them, and those systems change constantly, which is exactly why our guide to Google algorithm updates treats a core update as an ongoing re-weighting, not a one-time event. The broad factors Google names include:

• Meaning of the query. Google interprets what you're actually looking for, including synonyms, intent, and whether the query wants fresh results.
• Relevance of content. Whether the page's content matches the query, from basic keyword presence to deeper topical relevance.
• Quality of content. Signals of expertise, authoritativeness, and trustworthiness, including how other sites reference the page.
• Usability. Whether the page works well, mobile rendering, page experience, and loading behavior.
• Context and settings. Location, language, and device shape what's shown.

The levers you control directly are relevance, quality, and usability. You don't control the query, and you influence authority only indirectly over time. Google's Search Essentials is the closest thing to an official list of what to do and avoid at this stage.

What is PageRank, in plain terms?

A way of measuring a page's importance based on the links pointing to it. The original idea: a link is a vote, and a vote from an important page counts for more than a vote from an obscure one.

PageRank was one of Google's founding algorithms, and while ranking now draws on many systems, the underlying principle still holds, links carry authority, and authority flows through them. This is why internal linking matters: it distributes that authority across your own pages and helps Google understand which of your pages are most important.

It's also why a brand-new page with no internal or external links can sit unnoticed. Nothing is pointing to it, so nothing signals that it matters. Wiring a new post into your existing content is often the fastest, cheapest ranking lever available. Our technical SEO checklist includes the internal-link steps we run on every new page.

Where do AI Overviews and AI search fit in?

On top of the same index. AI Overviews and other AI search features draw on Google's existing index of crawled and indexed pages, they don't replace crawling and indexing, they sit above them.

The consequence is direct: if a page isn't crawled and indexed, it can't be pulled into an AI Overview, and it can't be cited. Everything upstream still has to work. What changes is the retrieval and presentation layer, where AI systems synthesize an answer from multiple sources rather than returning a ranked list of links.

That raises the bar on clarity and structure. Content that states an answer plainly, uses clear headings, and can be quoted as a self-contained passage is easier for an AI system to extract and attribute. We cover the specifics of making pages quotable in our AI SEO guide. The foundation, though, is the same three stages this article describes.

Which stage is your page failing at?

Work through these in order and stop at the first "no." Each answer points at a different fix, and diagnosing out of order wastes time.

1. Is the URL crawled? Run it through URL Inspection in Search Console — our Search Console guide covers every report worth checking. If it's not crawled: check robots.txt isn't blocking it, confirm the server returns 200, and make sure real internal links point to it. Don't touch content yet.
2. Is it indexed? If it's crawled but not indexed, look at duplication and canonicalization first, then a stray noindex, then content depth. This is where the most common problems hide.
3. Does the right URL rank at all? If it's indexed but nowhere in results, check that Google chose the canonical you intended and that the page actually targets the query. A different URL of yours may be ranking instead.
4. Is it ranking but low? Now, and only now, is this a serving problem. Work on relevance, content quality, internal links, and the authority signals in Search Essentials.
5. Does JavaScript hide any of the above? If your content, links, or canonical only appear after rendering, treat every step above as suspect until you confirm what Google sees after the render pass.

Most teams jump straight to step four because that's the fun part. The stuck pages are almost always failing at step one or two.

What does Google Search NOT do that people assume it does?

Several persistent myths waste real effort. Clearing them out is worth more than most "advanced" tactics — and when rankings genuinely fall, myths are the first thing to rule out. Our guide to diagnosing a traffic drop gives the fuller sequence for telling a real cause from a false alarm.

The myth	What actually happens	What to do instead
Submitting a URL guarantees indexing	Submission requests crawling. Google still decides whether to index.	Fix the quality and duplication reasons a page isn't selected.
The meta keywords tag helps ranking	Google ignores it for ranking. It has for many years.	Spend the effort on relevant, well-structured content.
robots.txt removes a page from Search	It blocks crawling. A blocked page can still be indexed from links, without its noindex being seen.	Allow crawling and serve noindex to remove a page.
More pages always mean more traffic	Low-value pages can dilute crawling and quality signals.	Publish pages that each earn their place.
Ranking is a fixed formula you can reverse-engineer	Ranking uses many systems and changes constantly.	Build genuine relevance, quality, and usability.

The robots.txt one causes the most accidental damage. Blocking a page there doesn't remove it, and if Google can't crawl the page, it never sees the noindex you added to remove it. If you want a page out, allow crawling and serve noindex. Never both. Google's Search Central documentation spells out the correct removal methods.

How should you use this understanding week to week?

Build one habit: before you try to fix a page, name the stage it's failing at. Crawling, indexing, or serving, decide which before you change anything, because the fix for each is different and the wrong fix costs days.

This week, take your three most important pages and run each through URL Inspection in Search Console. Note whether each is crawled, indexed, and serving the URL you intended. You'll usually find at least one surprise, and that surprise is your highest-value fix.