The worst defect a bot has arrives on a calendar, not in a diff.

Because nothing about it changed, which is exactly the problem. The three most common causes of bot decay are all invisible in a change log: the model underneath it gets retired and silently swapped, a fact it was told once stops being true, and a link someone typed into its instructions stops resolving.

None of these throw an error. The bot keeps answering. It just gets quietly worse, in a voice nobody chose.

01 / Cause 1Cause 1: the model gets retired, and the vendor picks the replacement

This is the only software defect we know of that arrives on a calendar rather than in a diff. You change nothing. A shutdown date passes. The model powering the bot is swapped for one the platform selected, and everything downstream shifts a little: the tone, the length, the willingness to guess.

There is no error to read and no diff to review.

We track this because we had to. Two retirement waves were announced to us for the platform we build on, one in 2026-08 and one in 2026-10, each naming specific models. So far so ordinary. Here is what happened when we tried to verify them.

We could not find a public primary source for either date. The announcement reached us by email. The vendor's public changelog spans the first shutdown date and carries no entry about it. No help-centre article names either date.

And two days after the first wave's date, the vendor's own interface was still offering models it was supposed to have retired. We checked by reading the live application bundle. It lists retired models as available, and it also lists models that the runtime rejects outright if you try to use them.

So the vendor's own product over-reports in BOTH directions, and there is no instrument available to a customer that can confirm a retirement before it lands.

The operational rule we adopted: plan for the date, and do not wait for a confirmation that has no surface to arrive on.

02 / The inversionThe counter-intuitive part: not choosing is the exposed position

The instinct is that pinning a bot to a model that is being retired is the risky state, and leaving it on the default is the safe one.

It is the other way round.

A bot with no model specified runs the platform default, and the platform default is precisely what auto-migration moves. It changes with no diff, no notice, and no record on your side. A bot pinned to a dying model at least has a date attached to its problem and a person who can act on it.

If you run a bot and nobody can tell you which model it uses, that is the finding.

03 / No fix at allAnd one that has no fix at all

Depending on how a bot was built, the platform refuses to accept a model setting on it:

422  llm is only allowed when bot type is PROMPT_BASED_BOT

Those bots are exposed to auto-migration with no lever available to anyone. We flag them as such, and we are careful never to write that up as a fixable problem, because a finding with no possible fix is exactly how a safety check gets switched off by the people it nags.

04 / Cause 2Cause 2: a fact was typed in, instead of referenced

This one is ours to own, because we did it.

A bot's instructions can either contain a fact ("book here: example.com/booking") or reference a variable that holds it. Typing it in is faster and it works perfectly ... until the fact changes.

At a domain rebrand, four live bots turned out to be carrying private copies of the same link. One of them was pointing at a hostname that no longer resolved at all, on a page reached by paid advertising traffic. Nobody edited those bots. Nobody broke them. The world moved and the copies did not.

The rule we now enforce with a checker rather than a reminder: never type a shared fact into a bot's instructions. A price, a date, an address, a link, an opening time. Reference the variable, so there is one place to change it and one place to be wrong.

There are two distinct failures here and it is worth separating them, because most tooling only catches the first:

• A link that is already dead. Easy to detect.
• A copy that will die next time. Invisible to a link checker, because it currently works.

The second is the one that bites, and it is only detectable by looking for literals where a reference should be.

05 / Cause 3Cause 3: the bot learned something and wrote nothing down

The subtlest of the three. A person tells a bot a fact in conversation. The bot uses it correctly, right then, and stores nothing.

A week later an automated step asks them for something they already provided, because the record never learned what the conversation knew. From the customer's side that is not a small annoyance. It reads as being ignored by a company that is not paying attention.

A bot that recognises a fact and writes nothing has not learned it. If your bot cannot update the customer record, it has a memory that lasts exactly as long as one conversation.

06 / Honest maintenanceWhat honest maintenance looks like

Not "we will keep an eye on it". Four specific things, all checkable:

1. A pinned model, or a written note explaining why it cannot be pinned.
2. Zero typed facts. Every shared value referenced, not copied.
3. A build identifier the bot can state. Ours carry a version marker so a live bot can be asked which build it is running. Otherwise you are debugging a thing you cannot identify.
4. A dated re-read of every claim it makes about prices, hours and availability. Those are the ones that get someone standing outside a locked door.

07 / Our withdrawalAnd a benchmark of ours we are currently withdrawing

We ran a comparison of several models on a live client bot's real instructions, and it produced clean, useful-looking numbers. Some of them were counter-intuitive in the way good findings are, including one where a newer, smaller model performed worst.

Then we found the flaw. The instructions we fed to the models were the RAW template, and a real bot's instructions are a template full of placeholders that get filled in at runtime. On the bot in question, the version we tested was 38% of the real thing. It never errored. It returned plausible tables measuring what those models do without our actual rules in front of them.

So those numbers are withdrawn until the re-run lands. The harness now resolves the placeholders, refuses to run if it cannot read them, and prints how many it substituted.

We are publishing that here for the same reason we publish everything else: a test record containing only wins is a brochure, and everyone reading it knows.