Getting an AI feature to work in a demo is not hard any more. Knowing it still works in June, on data nobody anticipated in January, is the actual engineering problem, and it is where most implementations quietly fail.
A traditional system either works or throws an error. An AI system has a third state: running normally and producing worse answers than it used to. Nothing alerts. The dashboard is green. This is the state that costs businesses money, and detecting it is a build decision you make before launch, not a thing you add later.
Build the test set first
Before we write the automation, we collect real examples with known correct answers. A few hundred invoices, documents, records, whatever the agent will handle, each with the answer a competent person would give.
Assembling that is tedious and it is the highest-value work in the project. It gives you a number. Not a feeling that it seems good, an actual accuracy figure you can put in front of a CFO, and a baseline that later versions get measured against.
Without it, every change is a guess. Somebody adjusts an instruction because one output looked wrong, and nobody can tell whether the overall system got better or worse.
Include the cases you expect to fail
A test set of clean, typical examples tells you almost nothing, because clean typical examples are the ones anything handles.
So we deliberately load it with the hard cases: the scanned document at an angle, the supplier with three spellings, the credit note that looks like an invoice, the record with a field used the wrong way round. Those are the examples that separate a system that will hold up from one that will embarrass you in month two.
When a real failure appears in production, it goes into the test set permanently. That way the same mistake cannot come back, which is the only reliable form of progress here.
Watch the shape of the output, not just the errors
In production you rarely have the right answer to compare against, so you monitor for change instead.
We track how often the agent refuses or escalates, and what proportion of cases fall into each category. Those numbers are stable when things are healthy, and they move before anyone notices a wrong result. A jump in refusals usually means the incoming data changed — a supplier switched their document format, a system started sending a new field. A drop, with nothing else to explain it, is the one to worry about: it often means the agent has started answering cases it used to hand over.
Sample, always
However good the automation is, a small share of real output gets reviewed by a person every week. Not because we distrust the numbers, but because a sample catches the category of failure the metrics were not designed to see.
This is cheap and it is the control most implementations skip. Ten minutes a week for a year is nothing against the cost of finding out in December that something has been subtly wrong since August.
Know what a mistake actually costs
How hard you work at accuracy should follow the consequence, not the ambition.
Categorising documents for a search index wrong is an inconvenience. Getting a compliance date wrong is a different kind of problem entirely. We set the accuracy target and the amount of human checking from what a mistake does, which is why some of our agents run with light oversight and others have a person on every case.
Ninety-five per cent accuracy is excellent or unacceptable depending entirely on what sits in the other five.
Why this is the credential
Anyone can show you a working demo. The question worth asking a partner is how they will know it has stopped working, and what happens then.
If the answer involves a test set, a baseline number and monitoring on the shape of the output, they have built these before. If it involves a careful prompt and confidence, they have built one demo.
