Technical SEO: Crawlability, Indexing and Site Health

Technical SEO is the work that makes a website easy for search engines to crawl, render, index, and trust, the plumbing beneath everything else you publish. This guide is for founders, marketers, and developers who own a site's organic growth and want to understand what actually keeps pages from ranking. It covers crawlability and indexing, JavaScript rendering, sitemaps, canonicals, site architecture, Core Web Vitals, mobile-first indexing, HTTPS, structured data, redirects, and how to audit all of it. No page ranks for content a crawler never reaches. Fix the foundation first, and the content and link work you do on top of it starts to pay.

The short version

• robots.txt controls crawling, not indexing. A URL you block there can still show up in results as a bare link, because Google knows it exists but never saw your instruction to drop it.
• Crawled does not mean indexed. Google crawls far more than it keeps, and it decides which pages earn a slot.
• To remove a page from the index, allow crawling and serve noindex. Never combine that with a robots.txt block.
• Core Web Vitals are a tiebreaker between comparable results, not a shortcut past weak content or missing relevance.
• A technical audit is a repeatable sequence: crawl the site, compare it to what's indexed, check how pages render, then fix status codes, canonicals, and architecture.

What is technical SEO, and why does it come first?

Technical SEO is everything that lets a search engine reach, read, and understand your pages before content quality or backlinks can count for anything. Think of it as three gates in a row: a crawler has to fetch the page, a renderer has to build it, and an indexer has to decide it's worth storing. Content and links only influence rankings after a page clears all three.

This is why we treat it as the foundation rather than an afterthought. A brilliant article on a page that returns a 503, blocks the crawler, or canonicalizes itself out of existence will never rank, not because the writing is weak, but because Google never gets a clean copy to evaluate. Google's own explanation of how Search works lays out the same crawl, render, index sequence. Our breakdown of how Google Search works walks through it step by step.

When technical SEO is not your bottleneck: if pages are indexed and loading fine but ranking on page three, the problem is usually relevance, depth, or authority, not plumbing. Don't spend a quarter shaving milliseconds off load time when the real gap is that your content doesn't answer the query.

What does crawlability actually mean, and how is it different from indexing?

Crawlability is whether a search engine can fetch your pages at all; indexing is whether it decides to store and rank them. They are separate stages, and confusing them causes more accidental damage than almost anything else in technical SEO.

robots.txt governs crawling. It tells bots which paths they may request — the same file increasingly discussed alongside AI crawlers, which our llms.txt guide covers in detail. It does not remove anything from the index. A URL you disallow in robots.txt can still appear in results, as a bare link with no description, if other pages link to it, because Google is aware the URL exists but was never allowed to fetch the page and read your instructions.

The noindex directive governs indexing. It's a signal in the page's HTML (<meta name="robots" content="noindex">) or an HTTP header that tells Google to keep the page out of results. Here's the trap: for Google to obey noindex, it has to crawl the page and read the tag. If you also block that page in robots.txt, the crawler never sees the noindex, and the page can linger in the index. If you want a page gone, allow crawling and serve noindex. Never both.

Why do crawled pages fail to get indexed?

Because crawling earns a page consideration, not a place. Google crawls more URLs than it ever indexes, and it filters aggressively for quality, duplication, and value. Seeing "Crawled, currently not indexed" or "Discovered, currently not indexed" in Search Console is common, and it usually points to one of a few causes.

• Thin or duplicative content. If the page adds nothing the index doesn't already have, Google often skips it.
• Near-duplicate pages competing. Faceted URLs, tracking parameters, and printer versions all dilute a single piece of content across many addresses.
• Weak internal linking. A page with no internal links looks unimportant, and Google treats it that way.
• Quality signals across the site. On a site with a lot of low-value pages, new pages inherit skepticism.

The fix is rarely technical alone. Strengthen the page, consolidate duplicates with canonicals, and give it internal links from relevant pages. Our on-page SEO checklist covers the content side of this.

How does Google render JavaScript, and when does it break SEO?

Google renders JavaScript, but it does so in a second pass that can lag behind the initial crawl, and that delay is where things break — our JavaScript SEO guide covers how to confirm what Google actually sees on a JS-heavy page. When Googlebot first fetches a page, it reads the raw HTML. Content that only appears after JavaScript executes has to wait for the rendering queue, and if critical content, links, or metadata depend entirely on client-side scripts, Google may index a thinner version of the page than users see.

The practical risk is not that JavaScript is invisible, it usually isn't, but that important signals arrive late or inconsistently. Internal links injected by scripts may not be discovered promptly. Titles and canonicals set by JavaScript can be missed if they conflict with the raw HTML.

What to do: serve your primary content, links, titles, and canonical tags in the server-rendered HTML wherever you can, through server-side rendering or static generation. Use client-side JavaScript for enhancement, not for the content that has to rank.

When not to worry: a mostly static marketing site or blog rarely has a rendering problem. Reach for SSR complexity only when your content genuinely depends on JavaScript to exist.

Do you still need an XML sitemap?

Yes for most sites, though a sitemap helps discovery rather than guaranteeing indexing. A sitemap.xml file lists the URLs you want crawled and can carry a last-modified date, which helps Google prioritize what to recrawl. It's most useful for large sites, new sites with few external links, and sites with pages that aren't well connected internally.

Keep it clean. A sitemap should list only canonical, indexable URLs that return 200. Including redirected, noindex, or blocked URLs sends mixed signals and wastes the crawler's attention. A single sitemap file caps at 50,000 URLs or 50MB uncompressed; past that, split it and reference the parts from a sitemap index file.

When a sitemap won't save you: it doesn't force indexing and it doesn't fix a page Google has judged low-value. If a URL is in your sitemap but stays unindexed, the answer is a better page and stronger internal links, not resubmitting the sitemap.

When should you index, noindex, or canonicalize a page?

Use the smallest tool that solves the actual problem, and never stack tools that fight each other. Duplication, thin content, and crawl waste each call for a different signal, and picking the wrong one is how pages disappear by accident. Here's the decision rule we apply.

• Index it when the page should rank and collect links. Leave it crawlable, with no canonical pointing elsewhere.
• Canonicalize it when the page is a near-duplicate of a stronger version and you want the equity consolidated. Keep it crawlable and add rel="canonical" pointing to the primary URL.
• Noindex it when the page must never appear in results but users still need it, think thank-you pages or internal search results. Allow crawling and serve noindex.
• Block it in robots.txt only when a whole path has no value and no links, and you want to save crawl budget, accepting that a stray URL may still surface as a bare link.

The signals compared side by side:

Signal	What it controls	Use it when
robots.txt disallow	Whether the page is crawled	An entire low-value path should not be fetched at all
noindex	Whether a crawled page is kept in the index	The page must stay live for users but out of search results
rel="canonical"	Which of several similar URLs is the primary one	Duplicate or near-duplicate pages should consolidate into one

A canonical is a hint, not a command, Google can pick a different URL if your signals conflict. Make it consistent: internal links, the sitemap, and the canonical tag should all point to the same primary version — the stakes are higher on a multi-region site, where our hreflang and multi-region SEO guide covers how canonical and hreflang signals need to agree instead of fight each other.

How does site architecture affect crawling and rankings?

A flat, well-linked architecture helps Google find pages fast and understand how they relate, and it distributes link equity to the pages that matter. If a crawler has to click through five layers to reach an important page, that page reads as unimportant, and it may get crawled rarely, if at all.

Keep important pages within a few clicks of the homepage. Group related content into clusters, with a pillar page linking out to supporting articles and each supporting article linking back. This is both a crawl-efficiency move and a relevance signal: Google uses your internal links and anchor text to infer what a page is about and how central it is.

Internal linking is the lever most sites underuse. New pages need inbound internal links to be discovered and to earn authority. When we publish, wiring a fresh post into the existing cluster is a deliberate step, not an accident. The complete SEO guide puts architecture in the context of the wider strategy.

What are Core Web Vitals, and how much do they matter?

Core Web Vitals are three metrics Google uses to measure real-world page experience, and they act as a tiebreaker rather than a primary ranking force. They matter, but relevance and content quality decide most results first. Google's documented "good" thresholds are set out on web.dev's Core Web Vitals reference.

Metric	What it measures	Google's "good" threshold
LCP (Largest Contentful Paint)	How fast the main content loads	2.5 seconds or less
INP (Interaction to Next Paint)	How quickly the page responds to input	200 milliseconds or less
CLS (Cumulative Layout Shift)	How much the layout jumps while loading	0.1 or less

Measure with field data from real users where you can, not just lab tools, a lab score on a fast laptop hides what mobile visitors on slow connections actually feel.

When not to obsess: if your vitals are already in the "good" range, further micro-optimization rarely moves rankings. Spend that effort on content and links instead. Chase vitals hard only when they're genuinely poor and hurting the experience.

What does mobile-first indexing require?

It requires that your mobile version contains the same content, structured data, and metadata as your desktop version, because Google predominantly uses the mobile page to index and rank. If your mobile site strips out content, hides links, or drops schema to look cleaner, you are effectively hiding those signals from Google.

Responsive design handles most of this automatically, since desktop and mobile share one HTML source. Problems tend to appear on separate mobile URLs or setups that serve trimmed content to phones. Check that headings, body copy, internal links, images with alt text, and structured data all survive on the mobile rendering.

Does HTTPS affect SEO?

Yes, HTTPS is a lightweight ranking signal and, more importantly, a baseline expectation for a trustworthy site. Browsers flag non-secure pages, and a broken or expired certificate can scare off users and interrupt crawling. The ranking nudge is small; the trust and security reasons are the real case for it.

Get the basics right: a valid certificate, all resources loaded over HTTPS to avoid mixed-content warnings, and a single consistent 301 redirect from HTTP to HTTPS. Don't leave both versions of the site reachable, that creates duplicate URLs and splits signals.

What does structured data do for you?

Structured data describes your page's meaning in a machine-readable format so Google can understand it precisely and, in some cases, show rich results. It doesn't directly raise rankings, but eligibility for enhancements like FAQ, article, or breadcrumb treatments can lift visibility and click-through. Use the vocabulary from Schema.org and follow Google's structured data guidelines for which types are supported.

Add schema in JSON-LD, and mark up only what's genuinely on the page. Marking up content users can't see, or claiming ratings you don't have, can trigger a manual action. Clear, honest structured data is also increasingly how AI answer engines parse and quote your pages, our AI SEO guide covers that shift.

When not to bother: don't add every schema type you can find. Mark up the things that map to a supported rich result or that clarify genuine ambiguity. Random schema adds maintenance without benefit.

How do redirects and status codes shape technical SEO?

Status codes are how your server tells crawlers what happened to a URL, and getting them wrong quietly bleeds authority and wastes crawl budget. The difference between a 301 and a 302, or between a real 404 and a soft one, changes how Google treats the page.

Code	Meaning	SEO handling
301	Permanent redirect	Use for permanent moves; passes signals to the new URL
302	Temporary redirect	Use only for genuinely temporary moves; keeps the original indexed
404 / 410	Not found / gone	Correct for removed pages; let dead URLs return these, don't mask them

Avoid long redirect chains, each hop slows crawling and can dilute signals. Point old URLs directly at the final destination. And don't redirect a removed page to the homepage as a habit; if there's no relevant replacement, a clean 404 is the honest answer and Google handles it fine.

How do you run a technical SEO audit?

A technical audit is a fixed sequence you can repeat, not a scavenger hunt. The goal is to compare three views of your site, what exists, what's crawlable, and what's indexed, and resolve every gap between them. Google's Search Essentials is the baseline every audit checks against.

Crawl and structure. Run a full crawl. Map depth, find orphan pages, flag broken internal links, and check redirect chains and status codes.

Indexing. Compare the crawl to Search Console's index coverage. Investigate every "crawled, not indexed" and "discovered, not indexed" cluster. Confirm important pages are indexed and unimportant ones aren't.

Rendering. Use URL Inspection to see the rendered HTML Google builds. Confirm your key content, links, and canonical tags survive rendering.

Duplication and canonicals. Look for duplicate titles, parameterized URLs, and pages canonicalizing to the wrong place.

Performance and mobile. Check Core Web Vitals with field data and confirm mobile parity.

The tool categories you'll use: a crawler (to see structure), Search Console (to see how Google sees you), a performance tool (for vitals), and a log analyzer for very large sites (to see what bots actually request). For the full step-by-step version, use our technical SEO checklist, see our technical SEO services guide if you'd rather have the audit and fixes handled for you, or start further back with how to choose a CMS that won't fight you on any of this if the platform itself is still an open question.

What are the most common technical SEO mistakes?

The most damaging mistakes are the quiet ones, the settings that silently keep pages out of results while everything looks fine on the surface. These are the ones we find most often.

• Blocking a page in robots.txt and expecting it to drop from the index. It won't, because Google can't read the noindex it can't crawl.
• A stray site-wide noindex left over from a staging environment. This deindexes the whole site and often goes unnoticed for weeks.
• Canonical tags pointing to the wrong URL, or every page canonicalizing to the homepage.
• Redirect chains and loops that slow crawling and waste budget.
• Orphan pages with no internal links, invisible to both crawlers and users.
• Mixed signals: a URL that's in the sitemap, blocked in robots.txt, and carrying a noindex all at once.

Most of these come from a config change no one documented. That's why an audit compares intent to reality rather than trusting either alone.

Where should you start this week?

The single habit worth building is a monthly reconciliation between what you think is indexed and what Google actually indexes, because technical SEO decays quietly, one deployment at a time, and the gaps only show up when you go looking. Everything else in this guide is downstream of that one check.

This week, open Search Console, run URL Inspection on your five most important pages, and confirm each is indexed, canonicalized to itself, and rendering the content you expect. If any of the five fails, you've found your highest-priority fix.