How we test an AI agent before it goes near a customer

We do not let an agent talk to a customer until it has passed a fixed test set of real cases, scored by someone who did not build it, at a pass bar agreed before the build started. Typically 50 to 100 cases, run again on every prompt change, with four specific failure modes checked every time. If it fails, it does not ship, and no amount of "but it works when I try it" changes that.

That paragraph is the answer. Now here is the whole method, because the industry norm is to demo an agent doing something impressive once and call it validated, and that is how you end up apologising to a customer.

Why a demo proves nothing

An agent that works in a demo is an agent that worked on the input the person demoing it chose. That is not a lie, it is just not evidence. Language models are non-deterministic: the same prompt can produce a different answer on Tuesday than it did on Monday, and a model update can change behaviour overnight without anyone telling you.

So the question is never "does it work". It is "how often does it work, on what, and how would we know if that changed". You cannot answer that by trying it. You answer it with a test set.

Building the test set

This is the unglamorous part and it is 80% of the value.

We pull 50 to 100 real cases from the client's actual history. Real tickets, real transcripts, real records, with names stripped. Not invented examples, because invented examples are always tidier than reality and tidiness is what you are trying to test against.

Then we weight the set deliberately:

• Roughly 60% ordinary. The boring middle of the distribution. If it fails here, nothing else matters.
• Roughly 25% awkward. Two questions in one message, a customer who changes their mind halfway, missing fields, an attachment that is a photo of a screen.
• Roughly 15% cases it must refuse. Requests that should escalate to a human, not be answered. This is the group everyone forgets and it is where the damage is.

Every case gets a written expected outcome before the agent ever sees it. Not a model answer, a description of what a correct response has to contain and what it must not. Writing those is where you discover the business does not actually agree on the right answer, which is a finding in itself and one worth having early.

The four failure modes we check every time

1. Confidently wrong. The answer is fluent, well structured, and false. This is the dangerous one because it survives a skim review. We check it by grading against a source, not against plausibility.

2. Refusing the wrong things. An agent that escalates everything is safe and useless. We track the refusal rate on cases it should have handled, and treat a high number as a failure, not caution.

3. Answering the wrong things. The inverse, and the one that ends up in a screenshot on LinkedIn. Any answer given on a must-refuse case is an automatic fail for the whole run. Not a deduction. A fail.

4. Drift under length. Agents that behave for three turns and lose the plot on turn nine. We test the long conversation specifically, because the short one always looks fine.

The pass bar, agreed before the build

Set the bar first or you will negotiate with yourself later. Ours, as a starting position:

The bar is a business decision, not a technical one, so the person who owns the customer relationship signs it, not the engineer. If 95% feels low, remember the honest comparison is not perfection. It is what your current process does on a Friday afternoon, which nobody has ever measured.

Who scores it

Not the person who built it. This is the single cheapest quality control available and almost nobody does it, because it feels like distrust. It is not. The builder knows what they intended, so they read the intention into the output. A second person reads what is actually there.

For volume we grade with a model as a first pass and have a human check every disagreement and a random sample of the agreements. Anthropic's guidance on developing tests and the OpenAI evals repo are both reasonable starting points if you want to build this yourself rather than buy it.

The case every test set should carry

In November 2022 a man named Jake Moffatt asked Air Canada's website chatbot about bereavement fares. The bot told him he could buy a ticket at full price and claim the discount back within 90 days. That was wrong. The airline's actual policy required the request before purchase, not after.

He bought the ticket. Air Canada refused the refund. In February 2024 the British Columbia Civil Resolution Tribunal found for him and ordered the airline to pay $812.02.

The part worth sitting with is the defence. Air Canada argued the chatbot was a separate legal entity, responsible for its own words. Tribunal member Christopher Rivers called that submission remarkable, and found the airline had not taken reasonable care to ensure its chatbot was accurate.

That is the argument for evals in one sentence. Not that the model was wrong, because models are wrong sometimes. That nobody had checked, and "the AI said it" is not a defence anyone accepts.

Run it against the four failure modes above and it is a textbook case of the first. The answer was fluent, well structured, specific about a number of days, and false. It would have survived a skim review. A test set of thirty real bereavement enquiries with their correct answers would have caught it in an afternoon, for a fraction of what the ruling cost, and orders of magnitude less than the coverage cost.

Two more for the must-refuse pile

A Chevrolet dealership's assistant was talked into agreeing to sell a Tahoe for one dollar, and confirmed it as "a legally binding offer, no takesies backsies". The screenshot passed twenty million views. DPD switched off part of its chatbot after a customer prompted it to swear at him and write poems criticising the company.

Neither is a model failure in the interesting sense. Both are cases nobody wrote a must-refuse test for. That is the third failure mode, and it is the one that ends up in a screenshot rather than a ledger.

The lesson across all three: a test set is not a gate you pass once. It is a ledger of every mistake anyone in your category has already made, so you cannot make it again. Ours carries Air Canada. Yours should too, and it costs you nothing to add it today.

What this costs

Building a first test set is roughly two days of work for a competent person who knows the business. Running it is minutes. Re-running it on every change is the whole reason it is worth building, because the second time you ship a prompt tweak that quietly breaks something, you will find out in five minutes instead of five weeks.

If that sounds like a lot of process for a chatbot, consider what you already do before a new hire answers a customer unsupervised. You would not skip that. An agent is a system with your name on it, deployed at a scale a person cannot reach, and the NIST AI Risk Management Framework is worth ten minutes if you need the formal language for a board.

The one thing to take away

Ask any vendor pitching you an agent for their test set and their pass bar. Not their demo. If they cannot produce both in writing, they have not tested it, they have tried it. Those are different words for a reason.