An AI Agent Gives a Wrong Answer: Stop Rules, Verification and Handoff Before You Touch the Prompt

When an AI agent has given a wrong answer, the order of operations is stop, reconstruct, contain, hand off to a person. Editing the prompt comes last, because anyone who edits before reconstructing is changing a system whose behaviour they cannot yet describe, and destroying the evidence while doing it.

What counts as a wrong answer worth stopping for?

Not every poor answer is an incident. The line runs through what the answer caused, not through how bad it sounds.

The critical row exists because it is not solved by a correction message. A commitment made and a price quoted are facts in front of the customer even when the system was wrong, and the bottom two rows do not justify a stop at all.

The first move is to stop, not to fix

Stopping here does not mean shutting down a server. It means a switch that takes the agent out of replying in the channel where the fault occurred and routes that traffic to a person.

• Disable automated replies in the affected channel only, if you can, and everywhere if you cannot.
• Do not delete the conversation, the message, or the wrong record. That is the evidence.
• Do not edit the prompt, the source documents, or the settings at this stage.
• Record the exact time of the stop, so the incident window has a boundary.
• Name who is handling it. An incident with no owner stays open.

The reason stopping precedes everything is plain: an agent that erred once for a given reason keeps erring for that reason until somebody stops it. Every extra minute lengthens the list of customers who will have to be contacted.

How do you reconstruct what actually happened in the run?

Reconstruction starts from the run record, never from the memory of whoever reported it. If the record does not exist, that is the first finding of the incident.

1. Find the run by message id or conversation id, not by an approximate timestamp.
2. Pull the exact input as it arrived, including what the user wrote before it.
3. Pull the context that was sent to the model: which documents were retrieved and in what order.
4. Pull the raw output, before post-processing, and the processed output as well.
5. Mark the first point at which either of them stopped matching reality.
6. Replay the same input against the same version and see whether the result repeats.

Step six separates two different worlds. A result that repeats is a stable bug you can fix and then test. A result that does not repeat points at sampling variation, a changed context, or a retrieval that returned something different, and each of those needs an entirely different fix.

Containment: what already left the building

The agent is stopped, but the answer already exists out in the world. Containment is the list of places it reached, not a feeling that the matter has been handled.

• Messages sent to the customer in any channel, including reminders scheduled for later.
• Records written into systems: tasks, statuses, fields that were updated.
• Documents produced, such as a proposal or a summary sent as a file.
• Downstream actions triggered by another automation listening to that same record.
• Other customers who received the same answer, if this is a pattern rather than one case.

The fourth item is the one that gets forgotten. A flow listening for a status change does not know the status was written in error, and it will carry on acting. Reviewing the chain that runs after the agent is part of containment, as it is in any complaint and escalation workflow.

Why is editing the prompt the last move?

Because it feels like a fix while mostly altering evidence. Once the prompt has changed, the conditions that produced the fault can no longer be replayed, and the question "is it fixed" becomes untestable.

There is a deeper reason too: not every wrong answer originates in the prompt. The same symptom can come from a source document that was updated, a retrieval that returned an irrelevant passage, an external tool that returned an error the model ignored, or an unusual input nobody anticipated. Changing the prompt in those cases adds a layer instead of removing a cause. Broader background on where the errors come from is in how to avoid AI mistakes and hallucinations.

What is left behind once the incident closes

• A one-page incident record: what happened, when, the cause, what was done, what is still open.
• A new test case holding the original input and the correct result, run before any future change.
• An updated stop rule, if it turned out the defined severity did not catch this case.
• A logging improvement, if reconstruction stalled because some field was never stored.
• An explicit decision on when the agent resumes, and who approves that.

The regression set is the real asset. After ten incidents you hold ten cases that failed once and must never fail again, and that is the only thing stopping each fix from becoming the source of the next fault. Running them automatically before a release also answers, immediately, whether the latest change brought an old problem back.

Sources

• Google SRE - Postmortem culture: learning from failure
• NIST AI Risk Management Framework