We Tested 8 AI Crawlers on Our Own Site. Access Was the Easy Part.

We pointed eight AI crawler user agents, plus Googlebot as a control, at our own site on 28 October 2026. All nine got a 200, the same byte count, and the same title as a desktop browser. By the usual definition of an AI crawler audit, we passed everything.

Then we measured what those crawlers actually received, and the more useful number showed up: on our marketing pages, between 95.6% and 97.7% of the bytes we hand an AI crawler are not content.

What we tested and how

We requested one published article, our guide to llms.txt and AI crawler control, with nine user agents, and compared the status code, the byte count and the title against the same request made as a desktop browser. The browser baseline was 215,787 bytes.

The method is deliberately plain, so you can repeat it on your own site in about five minutes.

curl -s -o /dev/null -w "%{http_code} %{size_download}\n" \
  -A "Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" \
  https://example.com/your-page

We ran that for GPTBot, OAI-SearchBot and ChatGPT-User from OpenAI, ClaudeBot, Claude-SearchBot and Claude-User from Anthropic, PerplexityBot and Perplexity-User from Perplexity, and Googlebot as a control. The requests and the measurements below were executed by an automated agent, not by hand, which is why the byte counts are exact rather than rounded.

Every crawler got the same page

All nine returned HTTP 200 with a byte count identical to the browser baseline and a matching title element. No blocks, no challenge pages, no truncated responses.

Crawler	Operator	Status	Bytes	Title matches
GPTBot	OpenAI	200	215,787	Yes
OAI-SearchBot	OpenAI	200	215,787	Yes
ChatGPT-User	OpenAI	200	215,787	Yes
ClaudeBot	Anthropic	200	215,787	Yes
Claude-SearchBot	Anthropic	200	215,787	Yes
Claude-User	Anthropic	200	215,787	Yes
PerplexityBot	Perplexity	200	215,787	Yes
Perplexity-User	Perplexity	200	215,787	Yes
Googlebot	Google	200	215,787	Yes

Our robots.txt is permissive by design: User-agent: * with Allow: /, disallowing only /api/ and /admin/. There are no AI-specific blocks. An llms.txt file is present and returns 200.

Why a passing access test proves less than it looks

A user-agent test proves one specific thing: that nothing on your stack is keyed to the crawler's name. That is worth knowing, because user-agent blocks are the most common way sites accidentally shut AI crawlers out, usually through a CDN rule nobody remembers adding.

It cannot prove the crawlers themselves are getting through. Several CDNs and firewalls verify bots by their published IP ranges, not by the string they send. A test like ours sends the right name from the wrong address, so a real crawler may be treated differently in either direction: blocked when our test passed, or allowed when our test was challenged.

So read the result narrowly. Nine passes means we have no name-based blocks. It does not mean every engine successfully fetched this page, and we are not going to claim it does.

What the crawlers actually downloaded

Readable text ran from 2.3% to 12.5% of the payload: 11.5% on articles, 4.7% on everything else. This is the part that changed what we plan to work on.

For each page, we fetched as GPTBot, stripped script, style and noscript blocks, removed the remaining tags, collapsed whitespace, and measured what was left.

Page	Raw KB	Text KB	Words	Text share
/pricing	448.9	10.1	1,762	2.3%
/product	413.2	15.8	2,729	3.8%
/ (home)	494.8	19.7	3,461	4.0%
/about	135.2	5.9	1,030	4.4%
/blog	674.8	60.5	10,897	9.0%
/blog/llms-txt	210.7	21.7	3,674	10.3%
/blog/ai-seo-guide	231.5	26.8	4,540	11.6%
/blog/content-research-tools	251.0	31.3	5,432	12.5%

The three articles average 11.5% readable text. The five non-article pages average 4.7%. The pricing page is the extreme case: 448.9 KB delivered to carry 10.1 KB of words, which is 1,762 words of actual pricing information wrapped in roughly 439 KB of something else.

Our articles are not especially lean. They just contain enough prose to dilute the overhead.

Where the rest of the payload comes from

This site is a React application rendered on the server. Every server-rendered React page ships two copies of itself in different forms: the HTML a reader sees, and a serialized payload the framework uses to make the page interactive without refetching. Add inlined CSS, font preloads, JSON-LD and the component tree, and the markup around your sentences grows faster than the sentences do.

None of this is a bug, and none of it is specific to our framework choice. If you run Next.js, Nuxt, Remix or SvelteKit, your ratios will look broadly like ours. A hand-written HTML page would invert them.

Does a low text share hurt AI visibility?

We do not know, and we are not going to pretend otherwise. We found no evidence that any engine penalises a page for its markup-to-text ratio, and we are not aware of a study that tests it.

What can be said more carefully is this. Retrieval systems extract text before they do anything else, and extraction is not free. Fetchers have timeouts and size limits. To take the one case that is documented rather than inferred, Google states that "Googlebot crawls the first 2MB of a supported file type, and the first 64MB of a PDF file". A page that spends 97% of its payload on non-content is spending a budget it did not have to spend, and the cost lands on whichever system is least patient.

That is reasoning from how retrieval works, not a measured result, and we have labelled it that way on purpose. If you see a confident percentage attached to this claim somewhere, ask what was measured and how many times.

Run this on your own site

Two checks, in this order, because the second one only matters if the first one passes.

1. Access. Request one real page with each crawler user agent and compare status, byte count and title against a browser request. Differences matter more than absolute numbers. A 403, a redirect somewhere else, or a response that is 10% the size of the browser's all point at the same place: a rule in your CDN or firewall.
2. Payload. Fetch as a crawler, strip script, style and noscript blocks, strip the remaining tags, and compare what is left to the raw size. Do it for a template, not a page, since every page built from the same template will behave the same way.

If access fails, stop and fix that. Nothing else in optimising for ChatGPT and other engines matters while a crawler cannot read the page at all. Access is the one part of this work that is genuinely pass or fail.

What we are changing, and what we are not

We are not rewriting our marketing pages to chase a ratio. There is no evidence that would pay, and rebuilding a working site against an unproven metric is how teams waste a quarter.

We are doing three smaller things. We added the nine-crawler access check to our standing AI SEO audit, because it costs five minutes and catches a failure mode that is otherwise invisible. We are recording the text share per template so we can tell whether a future release makes it worse. And we are treating the pricing page as the one worth looking at, not because 2.3% is a problem we can prove, but because 1,762 words of pricing detail buried in 449 KB is worth a second look on its own merits.

The broader point is about what an audit is for. Checking whether AI crawlers are blocked is the easy half, and it was the half ours stopped at until we ran this. Measuring what the crawlers receive once they are through takes another five minutes and tells you considerably more about your site.

GrowthHasten builds SEO and AI-visibility software, so we run these checks on ourselves before recommending them. This one earned its place.

Method and limits

Measured on 28 October 2026 against growthhasten.com. Access test: one request per crawler to a single published article, user agent strings taken from each operator's published documentation. Payload test: one request per page as GPTBot, with script, style, noscript and comment blocks removed before tag stripping.

Four limits worth stating. This is a user-agent test and cannot reproduce IP-range verification. It is a single site on a single framework, so the ratios are illustrative rather than representative. The text extraction is naive: it counts navigation, footer and cookie text as readable, which makes our percentages slightly generous rather than harsh. And we did not test Google-Extended, the token that governs whether your content can be used to ground Gemini and AI Overviews answers. If you run this yourself, add it.