An AI agent can be wrong before it answers

An AI agent I was working on once skipped the step that would have told it whether its next action made sense.

The agent worked across Slack, Gmail, and Airtable. When a new message arrived, it sometimes needed to fetch the full thread before deciding what to do next. In one production case, it moved ahead without retrieving the earlier messages.

Nothing crashed. The workflow continued. Later steps simply operated on context the agent had never read.

That incident changed how I thought about accuracy. For an agent working across several tools, the visible answer is only the end of the process. The system can fail while gathering context, choosing a tool, updating a record, or deciding that no action is needed. A fluent response does not tell you whether those earlier decisions were sound.

The answer is only the visible part

An assistant that only returns text can often be evaluated on the response it produces. A workflow agent leaves more behind. It reads conversations, chooses actions, changes application state, and may trigger work in other systems.

Each individual step may look reasonable even when the sequence is wrong.

In the missed-thread case, the required behavior was simple: read the earlier messages before making a decision. The danger was that the agent already had enough information to keep going. Unless someone inspected the path it took, the missing context could remain invisible.

I began treating the path as part of the product output. Accuracy meant checking what context the agent retrieved, which tools it proposed using, what it would write to Airtable, what appeared in its work log, and what the user would eventually see.

That is a different quality bar from asking whether the final response sounds correct. It asks whether the system reached its result through a workflow we would trust it to repeat.

Turn production failures into evaluations

We changed the prompt so a final audit would catch the skipped thread fetch. I also added the incident to a small evaluation suite built from production patterns.

The suite had roughly fifteen scenarios. For each one, a model walked through the prompt as if it were the agent, and I reviewed the proposed actions. The missed thread became a case that every relevant prompt change had to face again.

The process was manual. The scenarios could not reproduce all the context of a live Slack conversation, and they could not prove that the agent was safe. Production monitoring still had to reveal failures we had not anticipated.

After that, the missed thread no longer depended on someone remembering it. It had a durable place in the product's quality bar.

Accuracy has to survive product change

Fixing the workflow created another problem. The old prompt and the revised prompt were supposed to behave differently. A permanent expected answer would preserve the old mistake. Replacing the expected answer whenever the prompt changed could make a regression disappear.

I used a current baseline for each scenario. It recorded the behavior we considered acceptable and the evidence supporting that judgment, such as the product specification, an explicit human decision, or a production outcome we had verified.

When a new prompt behaved differently, the baseline did not move because the result looked reasonable. The change needed an explanation. For the missed-thread case, the revised behavior could become the new baseline only after the audit requirement had been deployed and verified.

At that point, accuracy became a product decision. A model could compare the outputs. Someone still had to own the judgment that the changed behavior was better, and the final answer could not reveal every failure that happened on the way there.

I now look at two things: the result and the path that produced it. If the path is not trustworthy enough to run again, one good-looking answer is not evidence of an accurate system.