The AI answer that felt right was wrong

I was testing an AI product that investigated customer data and answered product questions. To compare two possible default models, I ran both several times.

The question I was evaluating: For users who started a subscription trial, how long did it usually take them to become paying customers?

I'll call them Model A and Model B.

• Model A, our previous default, said roughly one month. Its successful runs gave us similar answers.
• Model B, a newer and faster model, said closer to three months. Its answers varied more.

I chose Model A. Its answer matched the length of the trial and it appeared more consistent.

By the following afternoon, after checking the underlying data myself, I had reversed the decision.

Model B had been closer to the truth.

The mistake was treating a repeatable answer as evidence that the answer was right.

Why Model A won

I began the comparison after noticing a sudden change while I was working through the product's analyses. Responses had become much faster, but some of the results were harder to reconcile. I traced the shift to Model B replacing Model A as the product's default.

I ran the conversion-time question several times against each model.

Model A failed on two runs, but every successful run landed around one month. Model B often produced a figure closer to three months, with more variation between runs.

Model A seemed like the safer choice. Its answer was more stable and matched how I understood the product: if the trial lasted about a month, conversion around the end of that month felt reasonable. I chose Model A, and we changed the default back.

The comparison appeared sensible. I had considered speed, successful completion, consistency, and whether the result passed a basic sense check.

But I had no independently verified answer. I was comparing the models with each other and with my intuition.

What Model A had actually measured

I wanted an answer we could establish directly. I exported the underlying signup cohort, lined up the available trial and conversion timestamps, and started checking the analysis outside the agent.

I first suspected a problem with missing timestamps. The underlying records did not support that explanation.

The problem was in Model A's query.

I wanted to select users who had signed up during a reporting period, then follow those users for as long as it took them to convert. Model A required the signup, trial, and conversion to all happen inside the original reporting period.

That difference matters for anyone who signs up near the end of the period. If the user converts after it closes, Model A excludes the conversion. The later someone signs up, the less time the query allows them to become a customer.

The query systematically removed slower conversions and pulled the average toward one month.

Model B had allowed users to convert after the signup period. Its outputs were less consistent, but its query was closer to the question we meant to ask. The independently checked data supported the longer estimate.

I reversed my decision, and we switched the default back to Model B.

Why the repeated tests misled us

My repeated tests showed that Model A could produce the same interpretation more than once. They did not establish that its interpretation was valid.

Model A looked reliable because its successful answers clustered around the same believable number. The query was repeatable. The mistake inside the query was repeatable too.

I had blurred three different questions:

1. Does the analysis complete?
2. Does it return a similar answer when repeated?
3. Does it answer the question we intended to ask?

The repeated runs helped with the first two. The third required us to inspect the query and check its result against the underlying records.

An obviously absurd answer usually attracts scrutiny. Model A's answer was more dangerous because it came with a ready-made explanation. The trial lasted about a month, so a one-month conversion time sounded right. The model's query and our review were shaped by the same assumption.

What I would do differently

I would now start this kind of comparison with a few product questions whose answers we have checked directly against the underlying records. That gives the models a reference point outside their own outputs.

For each question, I would define the cohort, denominator, time boundary, and treatment of later events before reviewing the model's answer. Otherwise two models can answer subtly different questions while appearing to disagree about the same one.

I would also keep the generated query and intermediate counts beside the final result. How many users entered the cohort? How many converted? Which records were excluded, and why? Those details make it much easier to see where two analyses diverge.

Only then would I compare completion, repeatability, analytical validity, speed, and the ability to show the work. Those qualities need separate judgments. A fast model can still be inconsistent. A consistent model can still answer the wrong question.

For the product itself, the review experience matters. A polished paragraph and a final number encourage people to judge the conclusion. Showing the cohort definition, filters, time windows, and query gives them a chance to judge the work. High-consequence questions should also carry a clear path to manual verification.

In the immediate term, we kept Model B and made a few prompt updates. Its variability remained an open problem. That choice applied only to this analytical workload. Model B's wider performance remained unproven.

A useful model comparison needs at least a few independently checked answers. Without them, consistency and familiarity can point a team in the same wrong direction.

Since then, the AI result that fits my expectations is the one I most want to inspect.