AI Crawlers and llms.txt: Controlling How AI Uses Your Content

AI crawlers are automated bots that fetch web pages on behalf of AI companies, either to train large language models or to pull sources into an AI-generated answer. This guide is for founders, marketers, and developers who want to decide, deliberately, how much of their content those bots can use. We cover the difference between training and retrieval crawlers, the major bots you will meet in your server logs, what robots.txt can and cannot do about them, and what the llms.txt file is actually for now that Google has put its own position on the record. Primary sources where they exist, no invented behavior, just the mechanics and the trade-offs you control.

The short version

• Blocking AI crawlers can quietly remove you from AI answers. The bot that reads your page to cite it is often a different bot from the one that trains a model, and a single careless line can shut out both.
• robots.txt is the only widely respected control today. It works per user-agent, well-behaved AI bots obey it, and it does nothing to a bot that chooses to ignore it.
• llms.txt does nothing for Google Search. Google's own documentation says Search ignores the file, and no other engine has confirmed reading it, so treat it as low-cost housekeeping, not a ranking lever.
• Google-Extended is a robots.txt token, not a crawler, and blocking it does not affect your Google Search rankings.
• Training access and answer-engine visibility are separate decisions. You can allow one and block the other.

What are AI crawlers, and how do they differ from search crawlers?

They are bots that fetch your pages for an AI system rather than for a traditional search index. A search crawler like Googlebot exists to build the index behind a results page. An AI crawler fetches content to train a model, to build a retrieval index for an AI answer engine, or to grab a page in real time while a user is asking a question.

The overlap is real and worth naming. AI Overviews, for instance, run on Google's ordinary search index rather than a separate AI crawl, which is why the fundamentals of how Google Search works still decide whether you show up there at all. Standalone products like ChatGPT and Perplexity are the ones that send their own distinct crawlers.

So the practical question is not "do I want to be crawled." You already are. The question is which specific bots you allow, and for which purpose.

What is the difference between training crawlers and retrieval crawlers?

Training crawlers feed a model; retrieval crawlers feed an answer. They belong to the same companies, often share infrastructure, and obey the same robots.txt file, but they exist for opposite reasons, and that difference should drive every decision you make here.

Training crawlers

These collect text to help train or refine a model. Blocking one keeps your content out of that training corpus, which is a content-ownership decision. It does not, on its own, remove you from the live answers that model produces, because those often come from a separate retrieval step.

Retrieval and answer crawlers

These index or fetch pages so the AI can cite them when it responds. Block one of these and you are asking not to be quoted or linked in that engine's answers. There is also a third, narrower type: user-triggered fetchers that grab a single page because a person pasted a link or asked the assistant to read it. Those are initiated by a human, so a robots rule may not apply to them at all.

Get this distinction wrong and you can do the opposite of what you intended. Blocking a training bot to "protect content" is defensible. Accidentally blocking the retrieval bot next to it removes you from the AI results you probably wanted.

Which AI crawlers should you actually know about?

A handful account for most of the traffic. The table below lists the major ones, what each is really doing, and the exact way to control it in robots.txt. Names and roles here are taken from the operators' own published documentation.

Crawler / token	What it is	How to control it
GPTBot (OpenAI)	Training crawler that collects content for OpenAI's models	User-agent: GPTBot then Disallow: / to opt out of training
OAI-SearchBot (OpenAI)	Indexes pages so they can surface in ChatGPT search answers	Separate token from GPTBot; blocking it removes you from ChatGPT search results
Google-Extended	A robots.txt token, not a distinct crawler, controlling use of content for Gemini training and grounding	User-agent: Google-Extended then Disallow: /; does not affect Google Search inclusion or ranking
ClaudeBot (Anthropic)	Training crawler for Claude	User-agent: ClaudeBot; note Anthropic also runs Claude-SearchBot and Claude-User
PerplexityBot	Indexes pages for Perplexity's answer engine	User-agent: PerplexityBot; user-triggered fetches are a separate category
CCBot (Common Crawl)	Open web crawler whose public dataset is widely reused for AI training	User-agent: CCBot then Disallow: /

Two things to read out of that table. First, several vendors now run more than one bot, so a single User-agent line rarely covers a whole company. Anthropic, for example, uses distinct tokens for training, for search indexing, and for user-triggered reads, and blocking one leaves the others untouched. Second, Google-Extended is the odd one out: it is a control token bolted onto Google's existing crawl, not a bot that shows up separately in your logs.

How does robots.txt control AI crawlers, and where does it fall short?

By naming each bot's user-agent and disallowing the paths you want to withhold. It is the same file, at the same location, with the same syntax you already use for search engines. You add a block per bot:

User-agent: GPTBot
Disallow: /

That is genuinely the whole mechanism for opting out of a well-behaved crawler. The major AI operators state that their bots respect robots.txt, so in practice these directives work.

The limits matter just as much. robots.txt is a request, not a wall: a bot that chooses to ignore it faces no technical barrier, and it does nothing to stop a person from pasting your text into a chatbot by hand. It also cannot claw back content that has already trained a model, since training is a past event. And it governs crawling, not what a model does with pages it fetched before you added the rule. If you need enforcement rather than a polite opt-out, that lives at the server or firewall level, not in this file.

What is llms.txt, and what problem is it trying to solve?

It is a proposed file that gives an LLM a curated map of your most useful content. In the words of the proposal, llms.txt is "a proposal to standardise on using an /llms.txt file to provide information to help LLMs use a website at inference time."

The problem it targets is context, not permission. A model cannot read your entire site inside a single prompt, and raw HTML is noisy: navigation, scripts, and markup crowd out the actual answer. The idea is that you publish a short markdown file at /llms.txt pointing to clean, LLM-readable versions of your key pages, so an assistant can find the good material quickly.

The format is deliberately simple. It is markdown, it opens with an H1 naming the site (the only required part), it usually carries a short blockquote summary, and it lists links under H2 headings. You can read the full spec at the official llms.txt proposal.

How is llms.txt different from robots.txt and sitemaps?

They answer three different questions. robots.txt says who may crawl what. A sitemap lists every URL you want indexed. llms.txt is a curated shortlist of the content you think a model should read, written for machine comprehension rather than exhaustive coverage.

File	What it is for	Who honors it today
robots.txt	Grants or denies crawler access, per user-agent	Established standard; respected by well-behaved crawlers, including major AI bots
sitemap.xml	Lists all indexable URLs to aid discovery	Established standard; supported by search engines
llms.txt	Points LLMs to a curated set of clean, readable pages at inference time	A proposal; Google Search states it ignores the file, and no other engine has confirmed using it

The cleanest way to hold it in your head: a sitemap is for discovery, robots.txt is for access, and llms.txt is a proposed reading list. The first two are load-bearing. The third is optional, ignored outright by Google Search, and unproven everywhere else.

Does anything actually read llms.txt?

Google Search does not, and Google says so itself. Its guidance on optimizing for generative AI features is direct: "You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them." On the file by name it adds: "It's completely fine if you decide to create and maintain LLMS.txt files (or other similar files) for other services or systems that use these files." The catch is the line after it: "Doing so will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them."

Read that boundary precisely, because it is narrower than most summaries of it. Google is describing Google Search, generative features included. It is not speaking for OpenAI, Anthropic, or Perplexity, and to our knowledge none of them has publicly confirmed that its production systems read llms.txt. So the answer splits: closed and negative for Google, still open everywhere else.

That makes the file a small bet on the engines that have said nothing, which is roughly how we treated it anyway. A tidy llms.txt pointing to your best pages costs an afternoon and carries no documented downside. What changed is the quality of the reason: the recommendation now rests on a published position instead of on nobody having said anything.

When it is not worth your time: if maintaining it would pull effort away from the fundamentals, skip it. A well-structured page with clear headings and a plain-language answer does more for AI visibility today than any llms.txt file, and it does so through channels that demonstrably work.

Should you allow or block AI crawlers?

It depends on whether the value of AI visibility outweighs the cost of your content being used, and those two things pull in opposite directions. Most publishers we work with land on a split decision rather than an all-or-nothing one. Use the framework below to reason it through per bot type.

1. Do you want to appear in AI answers? If yes, allow the retrieval and search bots (OAI-SearchBot, PerplexityBot, Claude-SearchBot). Blocking these is the fastest way to disappear from those engines.
2. Do you mind your content training models? If you do mind, block the training bots (GPTBot, ClaudeBot, CCBot, and the Google-Extended token) while leaving the retrieval bots allowed.
3. Is your content your product? Paywalled research, proprietary data, and original journalism are the strongest cases for blocking training crawlers, because the content itself is the asset you sell.
4. Is discovery your goal? For most marketing sites, SaaS blogs, and documentation, being read and cited is the whole point. Here the default leans toward allowing both types.

When NOT to block: do not reflexively block every AI bot to "protect" a marketing site. If your business depends on being found, blocking retrieval crawlers removes you from a growing surface of AI-driven discovery for no real gain, since that content was published to be seen. Blocking is a content-protection move, not a security measure, and it makes sense only when the content genuinely needs protecting.

How do you implement each choice in practice?

Everything happens in two files, and neither takes long. Start with robots.txt at your domain root, add one block per bot, and deploy.

To allow AI answers but opt out of training, disallow the training tokens and stay silent on (or explicitly allow) the retrieval ones. Silence means "allowed" in robots.txt, so you only need Disallow lines for what you want to withhold.

To block a company entirely, add a separate block for each of its tokens, because one line does not cover a multi-bot vendor. Confirm the exact current token names in the operator's own documentation before you write the rules, since vendors add and rename bots.

If you choose to add llms.txt, do it for the non-Google consumers, because Google Search has said it ignores the file. Publish a markdown file at /llms.txt with an H1 site name, a one-line summary, and links to your cleanest, most useful pages. Keep it short and keep it current. A stale reading list is worse than none.

After deploying, check that a real browser and Googlebot are not accidentally caught by a broad Disallow. Our technical SEO guide covers how to validate robots.txt without locking yourself out of search, which is the failure mode that does real damage.

How does this connect to AEO, GEO, and being cited?

Allowing the right crawlers is the entry ticket; being genuinely quotable is what earns the citation. Answer engine optimization (AEO) and generative engine optimization (GEO) are about structuring content so an AI can lift a clean, correct passage and attribute it to you. None of that matters if your robots.txt blocks the bot that would have read the page.

So the sequence is simple. First, let the retrieval and search bots in. Second, write pages that state the answer plainly, use clear headings, and can be quoted as self-contained passages. That second step is where most of the work lives, and we go deep on it in our AI SEO guide and in our breakdown of Google AI Overviews. Watch for a syndication blind spot too: if a syndicated copy of your page gets crawled first or ranks above the original, an AI answer engine can just as easily attribute the citation to the republisher instead of to you. Google's own Search Central documentation remains the reference for the crawl-and-index foundation both of those depend on.

Crawler access and content quality are not competing priorities. They are two halves of the same visibility, and teams that treat AI search as part of their broader SEO services rather than a separate project tend to get both halves right.

Build one habit: before you edit a robots rule, name the bot and its job. Training or retrieval, and what happens to your visibility if you block it. Decide that on purpose rather than copying a blocklist from a forum, because a pasted rule is how sites accidentally delete themselves from AI answers. This week, open your robots.txt, list every AI user-agent you currently allow or block, and confirm each line matches a decision you would actually stand behind.