Our llms.txt is 81,804 bytes. That is 266 lines, and it is roughly fifteen times the size of Vercel's. We went looking at other companies' files, found ours was among the largest, and concluded that we had written the wrong kind of file.
The format exists to give a machine a short map of a site. An 81 KB map is not a map.
What llms.txt is supposed to be
A plain-text Markdown file at the root of a site, giving an AI system a compact index: what this company is, and which pages matter. It is a proposed convention, not a standard that any search or AI engine has committed to supporting.
The design intent shows in the shape of the good files. Here is Vercel's, 5,397 bytes, opening in full:
# Vercel
> Vercel is a cloud platform for building, deploying, and scaling
> web applications and AI workloads.
Use this index to find machine-readable documentation and platform
resources. Follow the linked indexes when you need individual pages.A name, one sentence of identity, then a pointer to further indexes. Netlify's is 2,973 bytes and does the same thing: a heading, one sentence describing the platform, then sections that link onward rather than enumerating every page inline.
Both of those are readable in a single pass. That is the whole product.
Ours is an archive, not an index
Our file opens correctly. It is a heading, then one sentence: "Own the answer when AI gets asked, and the ranking when Google gets searched."
Then it keeps going for another 264 lines, with multi-sentence descriptions attached to each link. Nothing in it is wrong. It is simply doing a different job from the one the format was designed for, and at 81,804 bytes it is closer to a content dump than to a table of contents.
We did not notice until we measured other people's. That is the uncomfortable part: we publish SEO software, we wrote a guide to llms.txt and AI crawler control, and our own copy had quietly grown past the point of usefulness.
We are not the worst case
Two files we checked are substantially larger, and both come from companies that know what they are doing.
| File | Size | Shape |
|---|---|---|
| searchengineland.com | 1,788,905 bytes | "## List of Posts", then the posts |
| mailchimp.com | 526,893 bytes | 13,922 lines, headed "Complete Web Content Analysis" |
| growthhasten.com (ours) | 81,804 bytes | 266 lines, descriptions per link |
| stripe.com | 69,923 bytes | not read |
| vercel.com | 5,397 bytes | identity, then links to indexes |
| netlify.com | 2,973 bytes | identity, then sections |
| squarespace.com | 2,409 bytes | not read |
| brightedge.com | 1,097 bytes | not read |
Search Engine Land's file is 1.79 MB and its second heading is "List of Posts". Mailchimp's is headed "Complete Web Content Analysis". Those are archives, and they are 1,600 and 480 times the size of BrightEdge's file respectively.
We have marked the files we did not open. Size is not shape, and we are not going to infer what is inside a file we only measured.
Why size is the thing that matters here
This is reasoning, not a measured result, and we want to be clear about which it is.
A retrieval system reading llms.txt is trying to decide where to look next. Every byte that is not helping it decide is working against the file's only purpose. A 1.79 MB index is not an index, it is the thing the index was supposed to point at.
What we cannot tell you is whether any of this affects anything. There is no published evidence that a major AI engine reads llms.txt, no engine has committed to supporting it, and we have seen no study linking the file to citation rates. Anyone quoting you a percentage lift from llms.txt is describing a method they have not shown. The same caution applies here that we wrote about in how to tell which numbers a model produced.
So the honest case for keeping the file short is not "it will rank you". It is that the file costs nothing to write, it might help, and a bad version of it cannot help at all.
What we are changing
We are cutting ours down to an index: the identity line it already has, then links grouped by section, with a short label rather than a paragraph on each. We expect to land somewhere near Vercel's size.
If you are writing one, the pattern worth copying is the one the small files share:
- One sentence of identity first. Every good file in this set opens this way, and it is also the single most useful line for an engine summarising you.
- Link to indexes, not to everything. Vercel and Netlify both point onward rather than inlining the site.
- A label, not a paragraph, per link. This is where ours went wrong.
- Do not duplicate your sitemap. A sitemap lists URLs for a crawler already indexing you. This file is supposed to say what matters and why.
- Serve it as plain text. Check the content type, not just the status code.
A note on measuring other people's sites
This started as a wider survey and did not survive contact with the data. Two domains in our set returned a real llms.txt to a direct request and then returned a 403 or refused the connection on scripted runs minutes later. If we had trusted the first automated pass, we would have published that both sites had no llms.txt. One of them has the largest file we found.
That is why there are no adoption percentages in this article. A census is only as good as its worst request, and ours was not good enough to put a number on. The sizes above are from requests we confirmed individually.
If you run this kind of check, repeat it, space the requests out, and treat a single failed fetch as unknown rather than absent. The same discipline applies to crawler access testing, which we wrote up in our own crawler access test.
Method and limits
On 29 October 2026 we requested /llms.txt from a set of SEO, marketing and SaaS domains, following redirects, with an identified research user agent, and recorded status and byte length. Files described by shape above were opened and read. Files marked "not read" were measured only.
Three limits. Byte sizes are a single snapshot of files that change. Two domains gave inconsistent responses between runs, as described above, so this article reports no adoption rate. And the argument that smaller is better is reasoning from the format's purpose, not a measured outcome, because no outcome has been measured by anyone that we can find.
Questions
Frequently asked questions
What is llms.txt?
A plain-text file at the root of a site, written in Markdown, meant to give AI systems a short index of what the site is and which pages matter. It is a proposed convention, not a standard any search or AI engine has committed to supporting.
Do AI engines actually read llms.txt?
There is no published evidence that any major AI engine reads it, and no engine has committed to supporting it. Treat anyone quoting a citation-rate lift from llms.txt as making a claim they have not shown the method for.
How big should an llms.txt file be?
Small enough to read in one pass. The files we opened that behaved like an index ran from about 1 KB to 5 KB: a name, one sentence of what the company does, then links onward. The largest we measured was 1,788,905 bytes, which is an archive rather than a map. Ours was 81,804 bytes, which is also too big.
Is llms.txt a replacement for a sitemap?
No. A sitemap lists every URL for crawlers that are already indexing you. An llms.txt is supposed to state what matters and why, for a system deciding where to look first. Duplicating your sitemap into it defeats the purpose.
Written by
GrowthHasten Team
Editorial Team, GrowthHasten
Articles from the GrowthHasten editorial team, grounded in primary research, our own testing, and what analyzing real websites shows, fact-checked in-house.
View profile
