AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Retrieval Can Cost AI Agents Accuracy Even When Their Calculations Are Right

AI agents increasingly work with tool outputs too long to fit in their context windows, so software harnesses often save those outputs to files that agents search or read on demand. A study from researchers at the University of California, Riverside and UC Irvine asks where this retrieval approach costs accuracy. Its answer is that the loss often lies not in the computation but in which evidence the agent uses to answer.

The team used PHREEQC, a geochemistry simulator from the U.S. Geological Survey whose reports can reach 846.8k characters. Because the simulator is deterministic, the researchers could re-run all 11,007 simulator calls the agents made and check every tool result they received. Agents answered 158 verified multiple-choice questions under one change: the report either appeared inline or was delivered as a table of contents (TOC), from which the agent read selected line ranges.

The effect depended on the model. Compared with inline output, TOC changed accuracy by −11.3 percentage points for GPT-5.1 (95% interval −18.0 to −4.9) and −7.9 points for GPT-5.2 (−14.9 to −0.9). The GPT-5.2 difference did not survive correction for testing five models. Claude Opus 4.6, Claude Sonnet 4.6, and GPT-5.4 showed differences between −0.6 and +0.2 points, with intervals that included zero.

The authors attribute the loss to what they call evidence substitution. GPT-5.2 and GPT-5.1 computed the keyed value about as often under every interface. Under TOC, however, they skipped the report in 60.3% and 53.0% of trajectories, respectively, and answered from their own output files, which print more digits. In one example, GPT-5.1 faced a question whose report showed pH 2.751, the correct option B. Its own file showed 2.75065. Given only the TOC, the agent read that file, truncated the value to 2.750, and chose distractor A. Inline, it answered B, as it did when told to answer from the report.

When the simulation was held fixed by replaying GPT-5.2’s logged run, TOC still cost 9.4 points. GPT-5.4 shows how a net result can mislead. It substituted evidence as well, losing about ten points on that front, but computed the keyed value more often under retrieval, leaving its net change indistinguishable from zero. An instruction to answer from the report raised accuracy over plain TOC by 7.9 points for GPT-5.2, 6.3 for GPT-5.1, and 7.8 for GPT-5.4. An instruction asking only for correct rounding did not help.

The study has clear limits. It tests closed models, one simulator, and questions written by one domain expert, and it excluded 42 of 200 original items after verification. The authors do not establish why the GPT models substitute evidence while Claude Sonnet 4.6 rarely does. GPT-5.1 remained below its inline accuracy even with the instruction, and only one wording of each instruction was tested.

For developers, the authors suggest that output delivery be validated for each model and judged by where accuracy is lost, not only by final scores. Clearly naming the authoritative output, or inlining reports that fit, worked in these experiments. Whether those remedies generalize beyond this simulator and these models remains an open question, but the work shows how a deterministic tool can expose failures that final accuracy alone would hide.