Robots.txt: A Practical Guide to Controlling Crawling

A robots.txt file is a plain text file at the root of your site that tells search engine crawlers which paths they may and may not request. It controls crawling, not indexing, and that single distinction is the source of most accidental damage in technical SEO. This guide is for founders, marketers, and developers who manage a site and want to understand exactly what robots.txt does, what it cannot do, and how to change it without knocking pages out of search by mistake. It covers where the file lives, the directives it supports, how Google matches rules, the common use cases and their limits, the trap that keeps blocked pages in results, and how to test everything safely.

The short version

• robots.txt controls crawling, not indexing. A URL you block there can still appear in Google as a bare link if other pages link to it, because Google knows the URL exists but was never allowed to read your page.
• To remove a page from the index, allow crawling and serve a noindex signal. Blocking it in robots.txt prevents Google from ever seeing that signal, so the page can linger.
• Never block the CSS and JavaScript that Google needs to render your pages. That hides how the page actually looks and can hurt how it is assessed.
• Google ignores crawl-delay. Set crawl rate through Search Console and better site performance, not through the file.
• The file must sit at https://yourdomain.com/robots.txt. Anywhere else and crawlers will not find it.

What is a robots.txt file, and where does it live?

It is a plain text file that gives crawlers instructions about which URLs on your site they are allowed to fetch. Crawlers request it before they crawl the rest of the site, read the rules that apply to them, and behave accordingly. It is part of the Robots Exclusion Protocol, a long-standing convention that the major search engines respect voluntarily.

Location is not optional. The file has to sit at the root of the host, at https://yourdomain.com/robots.txt, and it applies only to that exact host and protocol. A file at /blog/robots.txt does nothing, and https://www.yourdomain.com/robots.txt is a separate file from the non-www version. If a subdomain like shop.yourdomain.com needs its own rules, it needs its own robots.txt.

The file is public. Anyone can read yours by visiting the URL, so it is not a place to hide sensitive paths. Listing /admin/ or /private-beta/ there tells the whole internet those paths exist. Where robots.txt fits in the wider picture is covered in our technical SEO guide, which walks through crawling, rendering, and indexing as one sequence.

Does robots.txt control crawling or indexing?

Crawling only. This is the distinction that matters more than any other, because getting it wrong produces the opposite of what people intend. Crawling is whether a search engine may fetch a page. Indexing is whether it decides to store that page and show it in results. robots.txt speaks to the first and says nothing about the second.

When you Disallow a path, you are telling crawlers not to request those URLs. You are not telling Google to remove them from search. If Google already knows a blocked URL exists, usually because another page links to it, it can still list that URL in results, often as a bare link with no description, because it was never allowed to fetch the page and read what is on it.

To keep a page out of the index, you use a noindex signal, which lives in the page itself as <meta name="robots" content="noindex"> or as an HTTP header. For a fuller model of how these stages fit together, our guide to how Google Search works separates crawling, indexing, and ranking so you can tell which stage a page is stuck at.

What directives does robots.txt support?

Four directives cover almost everything you will ever need: User-agent, Disallow, Allow, and Sitemap. Each line is a single instruction, and rules are grouped under the crawler they apply to. The table below is a working reference.

Directive	What it does	Example
User-agent	Names the crawler that the rules below it apply to. * means all crawlers.	User-agent: Googlebot
Disallow	Tells the named crawler not to fetch URLs that start with the given path.	Disallow: /admin/
Allow	Carves an exception out of a broader Disallow, permitting a specific path.	Allow: /admin/help/
Sitemap	Points crawlers to your XML sitemap. Uses a full absolute URL, not a relative path.	Sitemap: https://site.com/sitemap.xml

Two supporting characters matter for matching. The wildcard * stands for any sequence of characters, and $ marks the end of a URL. So Disallow: /*.pdf$ blocks every URL that ends in .pdf. The Sitemap directive is independent of any User-agent group and can point to the sitemap that our XML sitemaps guide explains how to build and keep clean.

What does a simple robots.txt file look like?

A working file is short and readable. Here is a small example that allows most crawling, blocks two low-value paths, keeps an exception open, and points to the sitemap.

User-agent: *
Disallow: /admin/
Disallow: /cart/
Allow: /admin/public-docs/

User-agent: Googlebot
Disallow: /internal-search/

Sitemap: https://example.com/sitemap.xml

Read it top to bottom. The first group applies to all crawlers: stay out of /admin/ and /cart/, but /admin/public-docs/ is fine. The second group is specific to Googlebot and adds a block on internal search result pages. The Sitemap line sits on its own and applies regardless of crawler. Blank lines separate groups and have no other effect.

How does Google match rules and decide precedence?

Google applies the most specific matching rule, measured by the number of characters in the path, and a crawler only obeys the single group that best matches its name. This is where behavior surprises people, so it is worth being precise about both halves.

First, group selection. A crawler reads the User-agent lines and follows the most specific group that names it. If a file has a group for Googlebot and a separate group for *, Googlebot follows only the Googlebot group and ignores the wildcard group entirely. It does not merge the two. This trips up people who put general rules under * and expect every crawler to inherit them.

Second, rule precedence within a group. When both an Allow and a Disallow could match a URL, Google follows the one with the longer, more specific path. Given Disallow: /folder/ and Allow: /folder/page.html, the more specific Allow wins and that one page stays crawlable. Order in the file does not decide the outcome; specificity does.

Does Google support crawl-delay?

No. Google ignores the crawl-delay directive entirely, even though some other search engines honor it. Adding it does nothing on Google and can give a false sense that you have throttled crawling when you have not.

If Googlebot is crawling hard enough to strain your server, the levers that actually work are different. Improve server response times, fix pages that return errors under load, and use the crawl rate settings and reports in Search Console. In most cases the crawl rate self-adjusts once the server responds reliably, so the durable fix is performance, not a directive the crawler skips.

What is robots.txt good for, and where are its limits?

It is good for keeping crawlers away from paths that have no business being fetched, which saves crawl budget and server load on large sites. The strongest use cases share a shape: whole sections of low-value or non-public URLs that you also do not need to remove from an index they were never in.

• Admin and account areas. Paths like /admin/, /cart/, and /checkout/ that users need but search engines do not.
• Internal search results. On-site search URLs generate near-infinite low-value pages that waste crawl budget if left open.
• Faceted and filtered URLs. Filter and sort parameters on large catalogs can multiply into millions of near-duplicate URLs. Blocking the parameter patterns keeps crawlers focused on the real pages, though deciding which facet combinations deserve indexing in the first place comes before any directive.
• Staging paths on a live host. If a staging area shares the production host, blocking it stops crawlers wandering in.

The limits are real. On a small site under a few thousand URLs, crawl budget is rarely the constraint, so blocking paths buys little. robots.txt does not secure anything, does not remove pages from the index, and does not force Google to obey on a page it has already decided is important through links. For large or complex sites, deciding what to block is part of the work our SEO services handle as a considered audit rather than a guess.

Why can a blocked page still show up in Google?

Because a block stops Google from reading the page, not from knowing it exists. This is the single biggest trap with robots.txt, and it catches experienced teams. When a page is linked from elsewhere, Google records the URL. If that URL is disallowed, Google cannot fetch it to see the content or any noindex instruction, so it may index the bare URL on the strength of the links pointing at it.

The failure mode people cause most often is trying to deindex a page by blocking it. They add Disallow: /old-page/ expecting it to drop from search. Instead the page becomes uncrawlable, Google never sees the noindex tag they may have added, and the URL can sit in results for a long time. Blocking a page you want removed is exactly backwards.

The correct sequence: leave the page crawlable, serve a noindex signal, and wait for Google to recrawl and drop it. Only after it has left the index should you consider blocking the path, and usually there is no reason to. Allow crawling and serve noindex. Never both at once.

Why should you never block CSS and JavaScript?

Because Google renders your pages, and blocking the files it needs to render them means it sees a broken version of your site. Modern crawling includes a rendering step where Google executes CSS and JavaScript to build the page the way a browser would. If those resources are disallowed, the render fails or comes out incomplete.

The practical damage is that Google may judge a page on a stripped-down rendering: missing layout, missing content that scripts insert, and a mobile experience it cannot assess properly. Old robots.txt files that blocked /assets/, /scripts/, or /wp-includes/ are a common cause. Let crawlers fetch the resources required to render the page, and reserve blocks for actual URLs, not the machinery that draws them.

When should you not use robots.txt?

Do not use it whenever your real goal is to keep something out of search results, and do not use it to protect anything private. These are the two misuses that cause the most trouble, and both have better tools.

Do not use robots.txt to hide a page from search. Blocking prevents Google from seeing the noindex that would actually remove the page, so the URL can persist as a bare link. For deindexing, keep the page crawlable and serve noindex instead.

Do not use it as a security measure. The file is public and only lists paths you would rather people not know about. Anything genuinely sensitive belongs behind authentication or a server-level access rule, not a line in a text file everyone can read.

Do not block the resources needed to render a page. CSS, JavaScript, and images that build the layout should stay fetchable so Google can assess the page as users see it.

How do you test a robots.txt file?

Test before and after every change, because a one-character mistake in this file can hide an entire site. The good news is that testing is quick and the tools are free. Start in Google Search Console, which reports the robots.txt it last fetched, flags parse errors, and shows the fetch status for your property.

Then confirm behavior on the pages that matter. Run URL Inspection in Search Console on a URL you expect to be crawlable and on one you expect to be blocked, and check that each reports what you intended. Inspection tells you whether Google considers a specific URL allowed, which is more reliable than reading the rules in your head.

Two habits prevent most disasters. Fetch https://yourdomain.com/robots.txt in a browser after any deploy to confirm the live file is the one you wrote, and never ship a change straight to production without checking a blocked and an allowed URL. The full sequence sits inside our technical SEO checklist, alongside the other checks a healthy site needs. Google's own Search Central documentation gives the authoritative rules for how it parses the file.

What robots.txt mistakes cause the most damage?

The worst mistakes are silent: the file looks fine, nothing errors, and pages quietly vanish from search. These are the ones worth memorizing, because each one is easy to make and slow to notice.

• Blocking the whole site. A stray Disallow: / under User-agent: *, often left over from staging, tells every crawler to stay out of everything.
• Blocking a page to deindex it. The page stays in results as a bare link because Google never sees the noindex it cannot crawl.
• Blocking CSS and JavaScript. Google renders a broken version of the page and assesses it that way.
• Wrong location. A file anywhere but the host root is ignored, and www and non-www are separate files.
• A relative sitemap path. The Sitemap line needs a full absolute URL, not /sitemap.xml.
• Assuming inheritance across groups. A crawler with its own named group ignores the * group completely, so shared rules are not shared.

Most of these trace back to a change no one tested against a live URL. Google's Search Essentials is the baseline to check your intent against whenever something looks off.

The habit worth building is treating robots.txt as production code: never edit it in place on a live site without testing the change against a blocked URL and an allowed URL first. This file has more power to remove your site from search than almost anything else you can touch, and it fails quietly, so the discipline of testing every change is what keeps a one-line typo from costing weeks of traffic.

This week, open your live robots.txt in a browser, read it line by line, and run URL Inspection on your three most important pages to confirm each is crawlable. If anything you rely on is blocked, or if you find a Disallow: / hiding in there, you have just found your highest-priority fix.