We read the robots.txt of 28 SEO, marketing and SaaS companies on 29 October 2026 and checked what each one says about 14 named AI crawlers. Two sites block any of them outright. The other 26 do not.
The more striking number is what nobody says. GPTBot is the oldest of the named AI crawlers and the one most robots.txt guidance addresses, and 21 of the 28 do not mention it at all.
Almost nobody is blocking
Only two sites in the sample disallow a named AI crawler at the root.
| Site | Blocks outright |
|---|---|
| figma.com | GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider |
| notion.com | Amazonbot, meta-externalagent |
Figma's file is the clearest statement of intent in the sample. It blocks the training crawlers while leaving the retrieval agents alone: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User and PerplexityBot are all named and not blocked. That is a deliberate position, and a coherent one. Do not train on us, but do cite us when a user asks.
Most companies have not written a rule at all
Across the 28 reachable files, here is how often each agent is even named.
| Crawler | Blocked | Named, not blocked | Not named |
|---|---|---|---|
| GPTBot | 1 | 6 | 21 |
| Google-Extended | 1 | 5 | 22 |
| OAI-SearchBot | 0 | 5 | 23 |
| ChatGPT-User | 0 | 5 | 23 |
| PerplexityBot | 0 | 5 | 23 |
| ClaudeBot | 1 | 4 | 23 |
| CCBot | 1 | 3 | 24 |
| Applebot-Extended | 0 | 2 | 26 |
| Claude-SearchBot | 0 | 2 | 26 |
| Claude-User | 0 | 2 | 26 |
| Bytespider | 1 | 1 | 26 |
| Amazonbot | 1 | 1 | 26 |
| meta-externalagent | 1 | 1 | 26 |
| Perplexity-User | 0 | 1 | 27 |
Being unnamed is not the same as being blocked. We checked the wildcard group on each file: 27 of the 28 have a User-agent: * group that does not disallow the root, and the remaining file has no wildcard group at all. Not one of the 28 blocks everything by default. So an unnamed AI crawler is allowed on 27 of these sites, and the practical outcome is that AI crawlers can read nearly all of this sample.
What the table measures is attention, not access. For most of these companies, AI crawler policy is not a decision that has been made and recorded. It is a decision that has not come up.
The same inattention shows up in the other file companies write for AI systems. We measured llms.txt across much of this sample and found ours had grown to 81 KB without anyone noticing, which is its own kind of neglect.
The handful who did decide
Eight sites name at least one AI crawler without blocking it, which means somebody opened the file on purpose.
- frase.io names nine, the widest coverage in the sample.
- zapier.com and squarespace.com name eight each. Zapier's list includes ClaudeBot, Claude-User and Claude-SearchBot, so both Anthropic retrieval agents are explicitly accounted for.
- netlify.com names six and cloudflare.com names five.
- figma.com names five without blocking them, alongside the five it does block.
- moz.com and seoclarity.net name exactly one each.
Naming an agent and allowing it is not a no-op. It pins the behaviour so a later change to the wildcard group does not silently change your AI policy, and it documents that somebody thought about it. That is worth something on a file that several people will edit over several years.
What we would actually do
Allow the retrieval agents, and decide about the training crawlers separately. Those are two different jobs with two different consequences, and the question "should I block AI crawlers" hides the distinction.
Training crawlers such as GPTBot, Google-Extended, CCBot and Applebot-Extended take your content to improve a model. Google documents Google-Extended as controlling whether your content helps train and ground its generative models, separately from whether Google Search indexes you, in its crawler documentation. Blocking this group does not remove you from AI answers. Figma blocks these.
Retrieval agents such as OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User and PerplexityBot fetch your page to answer a question a user is asking right now, usually with a citation. Blocking these removes you from the answer. Figma does not block these.
Conflating the two is the common mistake, and it is expensive in one direction. If you block retrieval agents to protect your content from training, you have not protected anything and you have removed yourself from the surface where citations happen.
Our own file does not pass the test this article just set. It is a permissive wildcard group with two directories disallowed, a sitemap line, and not one AI crawler named:
User-Agent: *
Allow: /
Disallow: /api/
Disallow: /admin/So we allow everything, by default rather than by decision. We are in the 21, not the 7. We sell AI visibility measurement, so allowing both groups is the position we would have chosen anyway, but we had not written it down, and until we did it was an absence rather than a policy. We are adding the named groups.
Robots.txt is a request, not a wall
A disallow is a published preference that well-behaved crawlers honour. It is not enforcement. It does not stop a crawler that ignores it, and it does not stop content reaching a model through a third party that republished it.
It also does not tell you whether a crawler can actually reach your pages. A permissive robots.txt and a CDN rule that challenges non-browser traffic will happily coexist, and the second one wins. That gap is why we tested the crawlers against our own site rather than reading our own robots file and assuming. For what each directive does and how to write them, see our robots.txt guide.
Method and limits
On 29 October 2026 we requested https://<domain>/robots.txt for 30 named domains with an identified research user agent, following redirects, retrying up to three times. 28 returned a 200. The two that did not, searchengineland.com and webflow.com, refused our research agent on repeated attempts and are excluded from every count here. Each file was parsed into user-agent groups, and an agent counted as blocked only when its own group contained Disallow: / with no offsetting Allow rule. The full run was repeated and produced identical counts.
Four limits. The sample is 30 recognisable companies, not a random or representative one, so these are counts for this list rather than industry rates. Parsing robots.txt is less deterministic than it looks, since group merging and precedence vary between implementations, and a crawler may read a file differently from our parser. Sites can serve different robots files to different user agents. And this is a single snapshot of files that change.
Questions
Frequently asked questions
Should I block AI crawlers in robots.txt?
It depends which job you mean. Blocking training crawlers such as GPTBot and Google-Extended keeps your content out of model training. Blocking retrieval agents such as OAI-SearchBot and PerplexityBot removes you from the answers users see right now. Most sites want the second group allowed.
What is the difference between a training crawler and a retrieval agent?
A training crawler collects content to improve a model. A retrieval agent fetches your page to answer a question a user is asking at that moment, usually with a citation back to you. They use different user agents, so you can allow one and block the other.
Does blocking GPTBot remove me from ChatGPT answers?
Not on its own. GPTBot is the training crawler. The agents that fetch pages to answer live questions are OAI-SearchBot and ChatGPT-User, and they are named separately in robots.txt. Blocking GPTBot alone does not remove you from the answer surface.
Does robots.txt actually stop AI crawlers?
It is a published request that well-behaved crawlers honour, not enforcement. It does not stop a crawler that ignores it, and it does not stop your content reaching a model through a third party that republished it. It also says nothing about whether a crawler can reach your pages, since a CDN rule can block traffic a permissive robots.txt invites.
Written by
GrowthHasten Team
Editorial Team, GrowthHasten
Articles from the GrowthHasten editorial team, grounded in primary research, our own testing, and what analyzing real websites shows, fact-checked in-house.
View profile
