Should your site have an llms.txt? Probably not, and the reason is more useful than the answer.

Probably not, and the reason is more useful than the answer. When we measured a national register of 400 regulated businesses, we found that of the sites carrying one, at least six of the fourteen verified files were not written by the business at all... and every single AI-crawler block we found was a setting a CDN applied on the owner's behalf.

Nobody in that sector has opted out of AI crawling. Nobody has opted in either. Their vendors decided, and nobody told them.

01 / What it isWhat llms.txt is supposed to be

A plain text file at /llms.txt that tells AI systems what your site is about and which pages matter. The proposal is a couple of years old and it has been enthusiastically recommended by a lot of people who sell services adjacent to it.

The honest status: it is widely published and its consumption by retrieval systems is unproven. We have not found evidence that any production AI search system reads it to decide what to cite.

02 / Our withdrawalWe are withdrawing a number we used to repeat

This is the part we would rather not write.

Our own notes previously carried the widely-quoted figure that 97% of llms.txt files are never requested by anything. When we went to trace it to a primary source for this page, we could not retrieve one, so we withdrew it. It is no longer in our material and it will not appear in anything we send a client.

That does not make the claim false. It means we cannot stand behind it, and a statistic you cannot stand behind is a liability rather than an argument. We would rather publish the withdrawal than quietly stop using it.

03 / What we measuredWhat we measured instead

A defined, enumerable population: 400 services drawn from a national regulated register, in one country, in one sector, in 2026-09. The measurement spec was written and frozen BEFORE the first fetch, so nothing could be tuned afterwards to make the finding neater.

The first number is the denominator, and it is the one most studies hide. The register contains 98 columns of data and not a single website address. So every URL in the study was constructed by us, and that step is where coverage goes:

Read that as a property of OUR METHOD, not of those businesses. "Unresolved" means we found nothing. It does not mean the business has no website.

04 / The findingThe finding

21 of 109 sites served a file at /llms.txt. That is 19.3%, and it is wrong.

The number was three times higher than we expected, which is exactly the moment a result deserves to be distrusted rather than published. The numerator was small enough to check completely, so we opened all 21 by hand.

Six were not the business we were looking for at all. Four were online stores (including a record shop), one a speech-pathology practice, one an agricultural business. One more could not be identified.

Verified figure: 14 of 109, or 12.8%.

That correction points the way it does for a structural reason, and it is the most transferable thing in the study. Two major website builders generate an llms.txt automatically. So when our method mistakenly resolved onto an online store, that store was MORE likely to carry the file than a real business in the population was. Our errors pushed the number up. Publishing the raw 19.3% would have been publishing an artefact of our own resolver.

A study of this shape that sampled its numerator instead of checking all of it would have published 19.3% with a straight face.

05 / Who wrote themWho actually wrote those files

Of the 14 verified:

• 3 are website-builder platform defaults, each announcing the platform by name in the file itself.
• 3 are the same file with the brand name swapped ... three separately-branded businesses serving an identical marketing sentence. An agency template, replicated.
• 1 was generated by an SEO plugin, which says so in the file.
• 1 is clearly hand-written: real prose, real detail about the actual business, correct history.
• The rest are unattributed.

So at least 6 of 14 were not authored by the business whose site they sit on, and in a 400-business sample of a regulated national register we found exactly ONE file a human at the business appears to have written.

Quality, where a file existed: 16 of 21 opened with a valid heading. Of 63 links sampled from inside those files, 5 were already dead. Published once, never maintained. Which tells you what these files are: a checkbox, ticked, then forgotten.

06 / The other halfThe other half, which is more important

While we were there we read every robots.txt. 89 of 109 sites served one. Three of them block at least one AI crawler.

Then the part that reframes the whole topic:

All three blocks came from the same CDN's managed list. Every one carried the provider's own generated marker, and all three blocked the identical eight agents. Hand-written AI blocks: zero. Not one business in the sample made this decision itself.

Two details that matter if you are thinking about this seriously:

• None of them blocked the crawler that actually governs appearance in one major AI assistant's search answers. They blocked the training-only crawler from the same company. Those are different agents doing different jobs, and a business that thinks it has opted out of AI search has probably opted out of something else.
• The regulator that publishes the register does exactly the same thing on its own site: an AI-crawler block, inside the same CDN-generated block, that nobody there appears to have chosen.

We also found one site publishing an llms.txt that invites AI systems to read it, while its robots.txt blocks eight AI crawlers. The owner did not author the contradiction. One vendor published the invitation, another published the block, and nobody was in the middle.

07 / What to doSo what should you actually do?

Not "add an llms.txt". That is the recommendation this study does not support.

The useful version:

Whatever your website currently tells AI systems, you probably did not write it. You can find out in about thirty seconds.

Open two URLs on your own domain: yourdomain.com/robots.txt and yourdomain.com/llms.txt.

If there is an AI-crawler block in there you did not choose, that is a business decision your hosting provider made for you, and it is worth deciding on purpose. If there is an llms.txt describing your business in words nobody at your business wrote, same.

If both are empty, you are with the large majority: of the 109 sites, 14 carried a verified llms.txt and just 3 carried any AI-crawler rule at all. Having neither costs you nothing that we can demonstrate.

08 / Where it is weakWhere this measurement is weak

Stated as prominently as the findings, because the alternative is marketing wearing a lab coat.

• URL resolution is the dominant weakness. At 65% precision, about a third of the resolved set is not the population. We caught the effect only because we checked the numerator exhaustively.
• One market, one sector. This is not "small businesses" and we will not write it up as though it were.
• Static fetch only, one vantage point. A file injected by JavaScript, or a robots.txt that varies by CDN edge, is invisible to us. We think both are rare. We have not measured that.
• Small numbers where it matters most. Three blockers, fourteen verified files. The confidence intervals are wide and they are published in the full study rather than rounded away.

And one we are less comfortable with: our first crawl run failed at 325 of 400 sites and saved nothing, so the restart re-fetched hosts we had already visited. We used more of those sites' bandwidth than our published rate limit implies. That is fixed, and it is stated here rather than left out.