Your agent computed the right answer, then misread the report: GPT-5.1 loses 11.3 points when tool output moves out of context, Claude shows no detectable cost
A PHREEQC study of 158 geochemistry items names a new failure, "evidence substitution". Two cheap harness fixes recover 6.3–7.9 points.
Under retrieval they often skip the report, answer from their own higher-precision output files, and misround to a neighboring option, a failure we call evidence substitution.
Comprehensive take
Judge a harness change by where accuracy is lost, not just the final score. Log which evidence each answer used, and name the authoritative output in the prompt.
Use cases
Audit your own harness
Record whether each final answer cites the tool report or the model's own intermediate files; a flat accuracy curve can hide evidence-use loss.
Cheap mitigation
Name the authoritative output file in the prompt, or inline outputs that fit in context; the paper reports gains of 6.3–7.9 points for the GPT models.
Specs
| Benchmark items | 158 execution-verified multiple-choice items (PHREEQC) source |
|---|---|
| Largest report size | 846.8k characters source |
| Retrieval cost, GPT-5.1 / GPT-5.2 | 11.3 / 7.9 points source |
| Retrieval cost, Claude Opus 4.6 / Sonnet 4.6 | No detectable cost source |
| Logged simulator calls re-run | 11,007 source |
| Gain from naming authoritative output | +6.3 to +7.9 points (three GPT models) source |
Insights
- Axion reading: evidence substitution is a harness-observability problem. Teams that track only final accuracy will miss it.
- Axion caution: a single-domain preprint; treat the model ranking as a hypothesis to test on your own tools.
What the paper tested
Many agent harnesses now avoid flooding the context window with long tool output. Instead they write it to a file and let the model search or read it on demand. Zhang, Chundur, Owusu, Qomi and Raissi ask a narrow question: where does that choice cost accuracy? They use PHREEQC, the USGS geochemistry simulator, whose reports reach 846.8k characters and whose deterministic output lets every printed value be traced. Agents answer 158 execution-verified multiple-choice items with it (arXiv, 19 October 2026).
The numbers
When the report is read through a section index rather than placed in context, GPT-5.1 loses 11.3 points of accuracy and GPT-5.2 loses 7.9. Claude Opus 4.6 and Claude Sonnet 4.6 show no detectable cost. The authors re-ran all 11,007 logged simulator calls and found that both GPT models still compute the keyed value about as often as before. What changes is the evidence they answer from: they skip the report, read their own higher-precision output files, and round to a neighbouring option. Holding the simulation fixed, retrieval still costs GPT-5.2 9.4 points. GPT-5.4 is the instructive case: more frequent correct simulations mask an evidence-use loss of about ten points, so the net change looks like zero.
Why practitioners should care
The last result is the one to forward to a colleague. A flat headline accuracy number can hide a real regression, because a model that computes more correct answers can offset one that reads its evidence worse. If you only track end-to-end accuracy when you change how your harness handles tool output, you can ship a degradation and never see it.
The fixes are cheap. Naming the authoritative output raises accuracy by 6.3–7.9 points over plain retrieval for the three GPT models. For GPT-5.1 and GPT-5.2, inlining outputs that fit in context works as well, while simply asking for correct rounding does not.
Caveats
This is one tool, one domain and one multiple-choice format, from a single preprint that has not been peer reviewed. The abstract does not say which Claude or GPT versions are the newest available, and "no detectable cost" is a statement about statistical power on 158 items, not proof of zero effect. We read the abstract and arXiv record; we did not independently reproduce the runs.
Should you change anything?
If your agents consume large tool output from files, add a harness test that logs which source each final answer was taken from, not only whether it was right. If you run GPT-family models with retrieval over tool logs, try naming the authoritative output file in the prompt before anything more elaborate.