We audited our own chatbot detector. On the question our own headline asked, it was right once in eight.

Every chat-coverage number we have ever published rests on one rule: search a website's raw HTML for 31 chat-vendor patterns, and if any matches, that site has chat. It is what produced our scan of 40 Malaysian universities. Nobody had ever checked whether it works.

So we checked, and we wrote down first, before fetching a single site, that we would publish the result whatever it said.

It said this.

Read the first row as: when our rule says a site has chat, a visitor can find something roughly six times in ten. And when a site really does have a visible chat control, our rule notices it roughly six times in ten.

Read the third row as: as a detector of AI chatbots, the rule is noise. In the university sample it flagged eight sites and one of them had an assistant that identifies itself as AI. In the fresh sample it flagged sixteen and none did.

That third row matters more than the others, because "40 Malaysian Universities. 3 AI Chatbots." was our own headline.

01A third of the sample could not be graded at all

67 of 200 sites, 33.5%, were indeterminate, and that number is a finding, not a rounding error.

39 were unreachable. 9 served an error page. 4 hid behind a consent wall or bot check. 3 were blank. 2 refused our crawler in robots.txt and were never fetched. 2 were login screens. 1 was a parked domain for sale. And 7 had a floating control we could see but could not identify (an unlabelled avatar, a dot-grid button, a bare speech bubble), where the honest answer is that a visitor could not tell either, and our own ethics rule forbids clicking to find out.

An instrument that cannot answer for a third of its inputs has a third-sized hole in every study built on it. In a normal write-up those sites quietly vanish from the denominator and reappear as "we measured 134 sites", where a blind spot looks like a smaller sample.

02Why it fails, in two directions

The false positives are mostly one pattern. wa.me fired on 27 sites, the single most common "chatbot" detection in the whole run. On 8 of them a visitor could see no conversation route at all; the match was a WhatsApp icon in a footer social row, or in one case a share-this-page-to-WhatsApp button. Of the 23 firings a human could grade, 16 were a deep link to an app rather than a chat widget. Our own published article says, in its own body, that a WhatsApp link is "a link, not an assistant", while the rule that produced the article counts it as one.

The false negatives split three ways, and only one is fixable by adding patterns:

1. Vendors not on the list. Zoho SalesIQ, RetailCRM, Avada, NewOaks, and two more we had never heard of. Six sites.
2. Vendors on the list that the HTML never mentions. LiveChat, Freshchat and Intercom, all injected at runtime by a tag manager. Four sites, and invisible to any rule that does not run JavaScript.
3. Self-hosted first-party chat. chatbot.<their-domain>, chat.<their-domain>, app.<their-domain>. Four sites. No vendor pattern can ever catch this class, because there is no vendor.

03What we are correcting

Our university article named three AI chatbots. Under a stated definition, that the surface says it is AI before you speak to it, at least four institutions had one. The one we missed was running a self-hosted assistant on its own subdomain, which is the exact class no vendor pattern can ever catch, because there is no vendor in it. That is a positive finding about them and an error by us. They are not named here for the same reason none of the forty were named in the original: the mechanism is the useful part.

We also attributed one institution's widget to the wrong vendor. The rule matched a stale vendor string still sitting in the page source while a completely different product was actually running on the page. A detector that reads the source rather than the running page will believe the leftover string every time.

04The rule we now cannot argue with

Our research backlog forbids publishing named league tables of failing businesses. Until today that was a principle. It is now arithmetic.

At a 42% to 43% false-negative rate on visible chat, naming a business as "has no chat" is publishing a claim we can now compute the odds of being wrong about, and those odds are close to a coin flip. A GTM-injected widget becomes "they have no chat." A site that 403s our crawler becomes "they have no chat." A first-party assistant on their own subdomain becomes "they have no chat."

Aggregate reporting is not a courtesy. It is the only mode in which an instrument's own error rate is survivable. That is why every coverage number we publish is a percentage and never a list.

05What we are doing about it

Not fixing the rule and quietly re-running it. The frozen version stays published with these numbers attached. Any improved detector is a separate measurement reported beside this one, never a replacement, and that was written into the pre-registration before we knew we would need it.