Crawl Budget: How Google Decides What to Crawl

Crawl budget is the set of URLs Google can and wants to crawl on your site. On most sites it is not why a page is missing from search results, and Google says so in the opening paragraph of its own guide. This is written for the SEO manager or technical founder who has been handed "we have a crawl budget problem" and needs to establish whether that is true. It covers the mechanism, what Google changed when it rewrote this documentation on 22 July 2026, how to run the diagnosis on your own property, and why noindex costs you a crawl instead of saving one.

The short version

• Every site starts with the same default, conservative crawl capacity limit. Capacity is not something you earn by publishing more. Demand is the half that moves, and Google now names page quality and relevance compared with other sites among the things that move it.
• noindex does not save a crawl. Google still requests the page, then drops it once it reads the tag. That is the cost, not the control.
• The trigger is not only a page count. A large share of your URLs sitting in Discovered - currently not indexed is Google's own third condition, and nothing about that state requires a large site.
• The vocabulary changed. What Google's 2017 explainer called the crawl rate limit is now the crawl capacity limit, and the Search Console crawl rate setting that post referenced no longer exists.
• Below roughly ten thousand crawlable URLs, a page that will not index is almost always a quality, duplication, or discovery failure wearing a crawl budget costume. That threshold is our recommendation, not Google's position.

What is crawl budget?

The set of URLs Google can and wants to crawl. Google's crawl budget documentation, last updated 22 July 2026, splits that into two parts that move independently: the crawl capacity limit, which is what Google can fetch without overloading your server, and crawl demand, which is what Google wants to fetch.

Two definitions in that guide are easy to skim past, and both change how you scope the work. Google's crawling infrastructure treats a site as a unique hostname, so www.example.com and docs.example.com hold separate crawl budgets. And the capacity limit is shared across all of Google's crawlers, so heavy demand from one reduces what is available to the others.

Element	Crawl capacity limit	Crawl demand
What it is	A cap on the total time your server spends holding connections open for Google, counting both parallel connections and how long each is held. Google also calls it hostload.	How much Googlebot wants to fetch from your site in the first place.
What moves it	Crawl health. Stable or improving response times raise the limit. Slowdowns, 5xx responses, and rate-limiting signals such as HTTP 429 lower it. Google's own finite resources cap the top end.	Google states demand varies with a site's size, update frequency, page quality, and relevance compared with other sites.
What you influence	Server response time and error rate. There is no dial to turn.	Perceived inventory, popularity, and staleness. Google names perceived inventory as the factor you can control the most.

Two consequences follow, and they are the reason the folk model gets this wrong. Capacity is not a reward for size, because every site begins at the same conservative default and Google's systems raise it over time only when there is demand to crawl more and the site stays healthy. The demand half is partly a judgement about your site rather than an inventory of it, which is why "delete some URLs" is an incomplete answer to a crawl problem.

What changed in Google's July 2026 crawl budget documentation?

Four things, and the vocabulary is one of them. The current guide carries a last-updated date of 22 July 2026 on the page itself, matched by a last-modified header of Wed, 22 Jul 2026 18:28:48 GMT. The documentation also moved: the older /search/docs/crawling-indexing/large-site-managing-crawl-budget path now returns a 301 to /crawling/docs/crawl-budget, which is worth knowing because plenty of published guides still cite the old address.

Element	The 2017 explainer	Current doc, updated 2026-07-22
Name of the supply side	Crawl rate limit	Crawl capacity limit, also called hostload
What raises or lowers it	Crawl health, plus a crawl rate setting a site owner could use in Search Console	Crawl health, plus Google's own finite resources. The Search Console setting is gone.
Where every site starts	Not stated	"Every site starts with the same default, conservative crawl capacity limit."
Crawl demand factors	Popularity and staleness	Perceived inventory, popularity, and staleness as the set you can influence, inside demand that varies with site size, update frequency, page quality, and relevance compared with other sites
Who should care	"if a site has fewer than a few thousand URLs, most of the time it will be crawled efficiently"	Three conditions: a million or more pages changing weekly, ten thousand or more changing daily, or a large share of URLs in Discovered - currently not indexed

Google keeps the 2017 blog post online with a banner warning that some of its information may be outdated. Read it as history. It is where the phrase "the number of URLs Googlebot can and wants to crawl" comes from, and that framing survived the rewrite even though most of the surrounding detail did not.

One question the rewrite created deserves a flat answer, because it is now being asked directly: there is no new published crawl limit. Google states that every site starts at the same default conservative capacity and that its systems adjust from there. It does not publish what that default is, in requests, connections, or URLs. Any specific figure you see attached to it was not sourced from the documentation.

Two of the guides ranking for this query have not caught up, and the gap is about dates rather than competence. Semrush's crawl budget guide carries a June 2024 update; Yoast's carries March 2024. I opened both. Neither states the default conservative capacity limit, and neither lists page quality or relevance compared with other sites among the demand factors, which is the part that changes what you should actually do. Semrush also links the documentation URL that now redirects. To be fair about what is there: Semrush does state Google's position on noindex correctly, so that is not a gap on its page.

Does your site actually have a crawl budget problem?

Probably not. The guide's first paragraph disqualifies you on either of two tests: a site without a lot of rapidly changing pages, or one whose pages appear to get fetched on the day they publish, is told it does not need the guide at all. Between them those two tests rule out most sites, including most SaaS properties with a documentation surface and a blog.

Three conditions in the current doc describe who should keep reading:

1. Large sites: a million or more unique pages with content that changes moderately often, about once a week.
2. Medium or larger sites: ten thousand or more unique pages with very rapidly changing content, daily.
3. Sites with a large portion of their total URLs classified by Search Console as Discovered - currently not indexed.

The third condition is the one almost nobody applies, and it is the only one that is not a size. It is a coverage state, which means it can fire on a site of any scale. If Google knows about a large share of your URLs and keeps electing not to fetch them, that is a rationing signal regardless of whether you have a million pages. Google also notes that the numbers in the first two conditions are a rough estimate to help you classify your site, not exact thresholds.

Before touching anything, three questions in order. Each can end the investigation, which is the point. The cheapest possible outcome is discovering the problem lives somewhere else.

1. Are new pages crawled within a day or two of publishing? If yes, stop here. Google's own opening paragraph has ruled you out and nothing downstream will change that.
2. What share of your known URLs sit in Discovered - currently not indexed? Read the Pages report in Search Console as a proportion, not a raw count. A handful of URLs in that state is ordinary. A large fraction of your total is Google's third condition.
3. How many crawlable addresses does your site actually serve? Not published pages. Every filter combination, sort parameter, paginated slice, session identifier, and internal search result is an address Google can request, and that gap is usually where a surprising number comes from.

A worked example from our own property, where the honest reading matters more than the finding. On 31 August 2026 our sitemap listed 129 URLs. One of them, /blog/content-syndication, was published on 22 August and carried a <lastmod> entry in that sitemap. URL Inspection on 31 August returned a coverage state of URL is unknown to Google, never crawled, nine days after publication.

That is not a crawl budget problem, and calling it one would be exactly the mistake this article exists to prevent. A 129-URL site is not being rationed. What the verdict actually reports is that the page never entered the queue at all, so no amount of crawl efficiency work would have touched it. Sitemap inclusion is a hint, not discovery. What I've seen in practice: this stage, sitting upstream of any budget question at all, accounts for most of the pages teams believe are being starved of crawl.

How do you read the Crawl Stats report?

Start with host status, then crawl purpose, then response codes. The report sits behind Property settings in Search Console, covers a rolling 90 days, and is available only for root-level properties, which catches out anyone whose property is a URL prefix pointing at a subfolder.

Three groupings carry most of the diagnostic value:

• Host status: whether Google hit availability problems fetching your robots.txt, resolving DNS, or connecting to your server. A red status here makes everything below it unreliable, so read it first.
• By purpose: Google splits requests into discovery, meaning URLs it had not crawled before, and refresh, meaning recrawls. A site publishing steadily but showing almost no discovery traffic has a different problem from one where refresh is consuming everything.
• By response: the proportion of requests answered with 200, 301, 404, and 5xx. Redirects and errors are fetches you paid for and got nothing back from.

The report has one limit that matters, and it is the point where people read too much into it. Google describes the example URLs it lists as representative rather than comprehensive, and says explicitly that a URL not appearing does not mean it was never requested. So Crawl Stats will tell you the shape of the problem and rarely the specific URLs. When you need per-URL evidence, you have to see the crawl in your server logs instead.

Does noindex save crawl budget?

No. It costs one. Google's guide is unambiguous, in the section on blocking unimportant URLs: "Don't use noindex, as Google will still request, but then drop the page when it sees a noindex meta tag or header in the HTTP response, wasting crawling time."

The scope matters and it is where the confusion starts. Google is not saying noindex is a bad directive. It is saying noindex is the wrong instrument for a crawl problem. If a URL has to leave the index, noindex is the correct tool and it depends on the crawl happening. If the goal is to stop paying for the fetch, the tag cannot help, because the fetch is how Google learns the tag is there.

Where this site has published it wrong: two of our own posts offer noindex in a passage that frames the problem as crawl budget. I would rather correct that in print than edit it quietly, so here is what each one says and what the guidance should be.

• Our internal linking guide: its large-sites table names the problem as filter and sort parameters that "eat crawl budget and create duplicates", then offers rel="canonical", noindex, or crawl controls as the remedy. Listing noindex as a fix for a crawl budget problem contradicts Google's sentence above. The correct order for that row is consolidation first, a robots.txt disallow second, and noindex only for URLs that must stay crawlable for some other reason.
• Our programmatic SEO guide: under a heading that promises indexation and crawl budget at scale, it advises: "Serve noindex on variants that lack enough unique data to stand alone, and keep them crawlable so Google can read the tag." That sentence is correct as indexation advice, and it is the right way to keep a thin page out of the index. What is wrong is what the placement implies, because a reader under that heading takes it as a way to save crawl and it is the opposite. The heading is being narrowed to indexation, where the body already sits.

Our guide to facet and parameter URLs already states this rule correctly and quotes Google directly, so the position on this site was never in doubt, only inconsistently applied. The instrument that genuinely prevents a fetch is a disallow, and the directives that actually block a fetch covers what that does and does not achieve.

What actually reduces wasted crawling?

Three groups of actions, in the order Google's guide presents them, which is also roughly the order of return.

Manage your URL inventory: consolidate duplicate content so crawling concentrates on unique content rather than unique URLs. Block genuinely unimportant pages in robots.txt. Return 404 or 410 for permanently removed pages; Google calls a 404 a strong signal not to crawl that URL again, and note the asymmetry it points out: blocked URLs stay in the crawl queue much longer and get recrawled when the block is lifted. Eliminate soft 404s, which keep getting crawled. Keep sitemaps current and include <lastmod>. Avoid long redirect chains.

Make your pages efficient to load: faster server responses mean Google can read more of your site over the same connections. The under-used half of this is HTTP caching. Supporting 304 Not Modified tells Google to reuse its cached copy when a page has not changed, which saves your server bandwidth and Google's resources. The request still happens, so this trims the cost of a fetch rather than preventing one.

Debug: check whether the site had availability problems during crawling before concluding that demand is the constraint. Availability failures and rationing look similar in a coverage report and have nothing in common as fixes.

One limit Google states directly, and it kills the most common tactic: do not use robots.txt to temporarily reallocate crawl budget toward other pages. Google will not shift the freed-up budget elsewhere unless it is already hitting your capacity limit. Blocking is for pages you do not want crawled at all, not a lever for redistribution.

The two structures that generate the most wasted addresses on the sites we work with are filter interfaces and paginated URL sets, and both have their own handling rules that do not belong in a crawl budget decision. If you genuinely need more capacity rather than less waste, Google names two routes: add server resources when the constraint is your own capacity, which shows up as Hostload exceeded in URL Inspection, and improve content quality, since it says the resources allocated to a site factor in popularity, overall user value, content uniqueness, and serving capacity.

How many pages do you need before crawl budget matters?

Roughly ten thousand crawlable URLs, and that number is ours rather than Google's. Google does not publish a threshold in that form. Its own figures start higher and serve a different purpose: they classify who should read an advanced guide, and it says in the same breath that they are a rough estimate rather than exact thresholds.

We publish one number instead of a comfortable range for a practical reason. A range gives every team a reading of it that justifies the work they already wanted to do, and crawl budget work is expensive and easy to mistake for progress.

Two caveats sit on top of it. The first is what you count: crawlable addresses, not published pages. A site with 800 articles and a filter interface can serve tens of thousands of addresses, and Google's budget applies to the second number. The second is that the threshold is a filter, not a gate. The coverage-state condition overrides it in both directions, so a small site with most of its URLs stuck in Discovered - currently not indexed has the problem, and a large site whose pages are crawled the day they publish does not.

Which crawl problems look like budget problems but are not?

Most of them. This is the table worth keeping, because the misdiagnosis is expensive in the specific way that wastes a sprint.

Symptom	The crawl budget reading	More likely, and how to tell
New pages take weeks to appear	Google has run out of budget for the site	Nothing links to them. Run URL Inspection and look at referring URLs and last crawl date. URL is unknown to Google means it never entered the queue.
Deep items in a paginated set never index	Budget runs out before page seven	A canonical tag pointing every paginated page at page one, or no internal links to the deep items at all. Check the canonical before the crawl rate.
The page was crawled but is not indexed	Budget problem	Not a crawl problem at all. The fetch happened. This is a quality, duplication, or canonical decision made after the fetch.
Crawl requests jumped and rankings did not	We earned more budget	A template change or a new URL pattern started generating addresses. Google's 2017 post says an increased crawl rate will not necessarily lead to better positions, and calls crawling necessary for ranking without being a ranking signal.
Thousands of parameter URLs in the Pages report	Urgent crawl waste	Real waste, but binding only above the threshold. Below it this is untidy rather than costly, and worth fixing on those grounds.

During our technical audits the sequence that saves the most time is fixed: coverage state first, then internal links, then canonicals, then crawl volume. Crawl budget is the last hypothesis on that list because it is the only one on it that most sites cannot have. That ordering is part of the larger diagnostic layer set out in our technical SEO guide, where crawling sits next to rendering and indexing instead of standing on its own.

How do you tell whether the work moved anything?

Slowly, and on a trend rather than a reading. Google's wording is that its systems adjust the capacity limit over time when there is demand to crawl more and the site stays healthy. It does not publish an interval, so anyone quoting you a number of days for this invented it.

Two instruments, used for different questions. Crawl Stats answers whether the mix changed: fewer requests spent on redirects and errors, more on 200 responses, discovery traffic rising if you were publishing into silence. Its 90-day window means a change reads as a slope, not an event. The Pages report answers whether coverage moved, which is the outcome you actually wanted, and it is the slower of the two.

One honest limitation: if you did this work on a site that never had a crawl budget problem, both instruments will show something changed and neither will show more indexed pages. That is the expected result, not a measurement failure.

The habit worth building takes two minutes and settles most crawl budget arguments before they start: run URL Inspection on the page you believe is being starved, and read the coverage state before forming a theory. URL is unknown to Google, Discovered - currently not indexed, and Crawled - currently not indexed are three different problems with three different fixes, and only the middle one has anything to do with crawl budget.

This week, open the Pages report, express the Discovered - currently not indexed count as a share of your total known URLs, and write the number down. If it is small, close the tab and go improve your internal links. If it is large, you have Google's third condition and the work in this article is yours. Getting that ordering right across a whole site is the substance of the technical SEO work we run for technology companies.