The failure nobody builds for: your AI agent stops being useful and says nothing

AI systems rarely fail loudly. They return something plausible, the workflow ticks over, the dashboard stays green, and quality decays for weeks. Monitoring built for software that crashes will not catch it, because nothing crashes.

Every other article about AI reliability is about uptime. Uptime is the easy part, and it is not what kills these projects.

How do you know an AI agent has stopped working?

With normal software, breakage announces itself. A 500, a stack trace, a page that will not load. Somebody notices within the hour because something visibly stopped.

An AI system has a third state between working and broken, and almost nobody builds for it: still running, no longer useful.

The API returns 200. The record gets written. The summary is generic, or subtly wrong, or identical to yesterday's. Every green light in your stack stays green, because nothing in your stack has an opinion about whether the output was any good.

The way you find out is almost always the worst way: a user stops trusting it, does not tell you, and goes back to doing it by hand. Our own way of putting this, from a client conversation in May:

It'll erode trust in the early stages of adoption. Users are like, oh, this doesn't work. I'll just go back to how it is.

By the time that reaches you as feedback, it is not a bug report. It is an adoption problem, and adoption problems cost roughly ten times what bugs cost, because you are no longer fixing software. You are rebuilding confidence.

What do you alert on when there is no error?

You stop alerting on failure and start alerting on shape. Six signals, cheapest first. The first three are close to free and will catch most of it.

1. Volume

How many times did it run today, against a normal day. A silent stop is almost always a volume cliff, and a volume cliff is trivial to detect and almost nobody watches it.

If you implement one thing from this article, implement this. An alert when today's run count drops below half of the trailing average is an afternoon of work and it catches the majority of hard stops.

2. Output length distribution

If yesterday's summaries averaged 180 words and today's average 40, something upstream changed. You do not need to know what, you need to know that.

Length is a crude proxy for effort, and crude proxies you actually watch beat sophisticated metrics you do not.

3. Refusal and empty rate

The proportion of runs that returned nothing useful. Empty strings, hedges, "I don't have enough information to answer that".

This is the single highest-value number on the list and most teams never compute it, because a refusal is a successful API call and nothing flags it.

4. Downstream acceptance

How often a human edited or discarded the output.

This is the closest thing to a free quality metric you will ever get, because the humans are already generating the signal. Every time someone rewrites the draft your agent produced, that is a data point you are currently throwing away.

5. Token and cost per run

Cost is a proxy for behaviour. A run that suddenly costs half as much is usually doing half as much work, and a run that costs three times as much is usually stuck in a loop.

Cost alerts also have a political advantage: finance already cares about them, so they tend to survive.

6. The eval set, on a schedule

The only signal on the list that measures quality directly. More on it below.

Notice what is not here: uptime, error rate, latency. Keep all three. They just will not catch this class of failure, and a dashboard full of green uptime is exactly what makes people think they are covered.

Design for the failure before it happens

The reliability work that has actually saved us is not monitoring. It is the scaffolding underneath, decided before the build.

Our own rule, in the words we use with clients: three fail-safes for the edge cases, and more time spent scaffolding and researching before the build than most people budget for, specifically to find the what-ifs.

A worked example that is not hypothetical. We ingest meeting transcripts into our own operating system, which embeds them so they can be searched later. Sometimes the source API drops and returns a 403. Nothing in that chain is broken. A naive version of that pipeline would lose that day's data silently and permanently, and the only symptom would be a search result that never appears months later.

So the collection is spread across a window of time with retries, rather than being a single scheduled attempt. If the first pass fails, the second or third catches it. The system is built on the assumption that the things it depends on will be unavailable sometimes, because they will be.

That is the unglamorous version of AI reliability. Not a better model. Retries, windows, and a refusal to trust the happy path.

The three questions to ask before you ship

1. What does this do if the upstream system is unavailable? If the answer is "skips silently", you have designed a data loss bug.
2. What does this do if the model returns nonsense? If the answer is "writes it to the record", you have designed a trust bug.
3. What does this do if it runs twice? If the answer is "sends two emails", you have designed an embarrassment.

What does an eval set look like in production?

An eval set is a fixed collection of inputs with known good outputs. You run it against the live system on a schedule and compare the results over time.

Three things make the difference between an eval set that helps and one that becomes shelfware.

Build it from real failures, not from imagination. Every time the system gets something wrong in production, that case goes into the set. Your eval set becomes your incident history, made executable. This also means it starts small and gets more useful, rather than starting big and going stale.

Run it on a schedule, not on deploy. This is the important one. The whole point is that this class of failure arrives without a deploy. A model version changes, an upstream API adjusts its response format, someone edits a prompt in a UI, your data drifts. None of that touches your code, so none of it triggers your CI.

Keep it small enough to actually run. Thirty good cases you run nightly beat four hundred you ran once in March.

We wrote up the pre-launch version of this in how we test an AI agent before it goes near a customer. This piece is the sequel, and the difference between them is the difference between testing and monitoring. One asks whether it works. The other asks whether it still does.

Who gets paged when it drifts?

Here is where good intentions die in a business of 60 to 200 people. Nobody is on call at 2am for a summarisation agent, and pretending otherwise produces alerts that everyone learns to ignore within a fortnight.

What works instead:

• Alerts land in a channel, not an inbox. Each system gets its own, and whoever is allocated to it sees them. Inboxes are where alerts go to be archived.
• The alert carries the fix, not just the news. "Token use is at 80% of the ceiling. Diagnose?" with a button, and the cost of the fix shown before anyone approves it. An alert that requires you to open four tabs is an alert that waits until Monday.
• Acknowledgement is visible to everyone. When one person hits investigate, that state syndicates so nobody duplicates the work and nobody assumes somebody else has it. The second failure mode of alerting is two people fixing the same thing; the third is nobody, because each assumed the other.
• Let the system heal the boring cases itself, with approval. A meaningful share of AI production issues are a resource ceiling rather than a mystery, and that is exactly the category a system can diagnose and fix if you let it ask first.

That last point is the one to steal. Most of what wakes you up is quota, not chaos.

The Australian version of the on-call problem

One thing that never appears in overseas reliability writing, and which matters if your team is here.

An Australian business has genuinely long quiet windows. A Friday evening failure with a public holiday Monday is four days of nobody looking. Easter is five. The stretch between Christmas and mid-January, when most professional services businesses are effectively closed, is a fortnight in which a silent failure is guaranteed to go unnoticed.

Our own experience is that silent failures get discovered on the way back in, not while they are happening. That is not a discipline problem. It is a calendar.

Two cheap consequences:

• Run the eval set on a schedule that includes the days nobody is working. The failure does not take the long weekend off.
• Put a volume check on the first morning back. Tuesday after a long weekend, or the second week of January. That is when you will find out what happened.

And a harder point worth stating plainly. If your AI system touches personal information and it fails in a way that exposes data rather than merely degrading, the Notifiable Data Breaches scheme gives you 30 days to assess a suspected eligible breach. That clock does not care that it happened over the break, and it starts when you become aware, which is an argument for becoming aware sooner.

The ASD's guidance on AI supply chain risks and mitigations is worth reading alongside this, because a surprising share of silent failures originate in something upstream that you do not control.

The cheapest version of all of this

If you take one thing from this piece and you currently have no monitoring at all, do this today. It is an afternoon.

1. Count the runs per day and alert if the count drops below half of normal. Catches most silent stops.
2. Log the output length and eyeball the trend weekly. Catches most quality decay.
3. Ask the three people who use it whether they still trust it. Monthly, in person, not a survey.

Point three is not a joke and it is the highest-signal instrument you own. The people who quietly stopped using your AI system know exactly when it stopped being useful. They will tell you if you ask directly. They will never raise a ticket about it, because from their point of view nothing broke. It just was not worth the bother any more.

Frequently asked

Can we not just use our existing APM tooling?
Keep it, and add the shape signals above. Application performance monitoring answers "is it responding". It has no view on "is the response any good", which is the failure mode that matters here.

How often should the eval set run?
Nightly is a good default for most mid-market builds. Weekly is defensible for something low stakes. On deploy only is the trap, because these failures arrive without a deploy.

Who should own AI monitoring internally?
Whoever owns the system the agent writes into, which is usually RevOps rather than IT. The person who cares whether the CRM is right is the person who will notice the alert.

What is an acceptable refusal rate?
There is no universal number, and anyone quoting one is guessing. Measure your own for a fortnight, then alert on deviation from that baseline rather than on an absolute threshold.

Does this apply to Breeze and other native AI features?
The failure mode applies. The instrumentation mostly does not, because you cannot run an eval set against a product you cannot inspect. That is one of the real trade-offs between native tooling and a custom layer.

Where to go from here

If the thing you are actually worried about is your data rather than your monitoring, start with 6 CRM problems that only show up months after onboarding. If you are about to give an agent write access to your CRM, the guardrails are in how to deploy HubSpot AI agents without breaking your data.

The question worth asking in your next stand-up: if the AI thing you shipped last quarter stopped being useful three weeks ago, how would anybody know?

Sources