AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Accuracy Alone Can Mislead Decisions About Restarting AI Agents

When an AI coding agent appears to be stalling, should a supervising system stop it and start over? A pair of researchers at Intuit argues that the usual way of grading the tools that make this call, by how often they correctly flag failures, can miss what actually matters: whether the restart improves the chance of success. The question has practical stakes, since each attempt consumes a limited budget of model calls, and abandoning a run that would have finished wastes that budget.

The authors, Siddharth Jain and Venkat N Vedam, base their framework on a simple observation. Stopping an attempt forfeits whatever chance it still had of succeeding, and a clean restart receives only the calls that remain. In a mathematical model, they show that a failure monitor with identical accuracy can produce very different results depending on how likely a fresh attempt is to succeed. With a 90% true-positive rate, a 10% false-positive rate, and a 50% chance that the original run would have succeeded, the monitor causes a 5-percentage-point loss if restarts never succeed, no change at a 10% restart success rate, and a 35-point gain at 80%. The authors describe these as analytical controls, not measurements of any real detector.

The paper also proposes a statistical audit that reuses existing agent traces without new model calls. For each task, it compares the estimated success of continuing with that of restarting, using simultaneous binomial confidence intervals, and reports “restart,” “continue,” or “abstain,” the last when the evidence does not separate the two options.

The experiments used 194 executions of the mini-swe-agent harness on 19 SymPy issues from SWE-bench Verified, with an external test checked after every action. Under the model openai/gpt-5.6-luna, a 16-call cutoff within a 30-call budget appeared to add 2.29 percentage points in-sample, with an interval of 0.00 to 6.86. The retrospective audit traced that apparent gain to a single surviving failed attempt on one task. On the second model, deepseek/deepseek-v4-flash-0731, the same cutoff lowered success from 68.57% to 26.29%, a 42.29-point loss. Prospective cohorts tested on the first model did not reproduce the gain: across all 19 tasks, the difference was −0.21 points, with an interval of −4.21 to 2.53. Of the 103 live comparisons in the audit, none met the certification bar. A fresh-context reset eliminated one diagnostic error label but did not improve success, which was 0 of 20 attempts versus 1 of 20 at baseline, while raising mean cost by 56.7%. Recombining failed patches produced no passing solutions.

The limitations are substantial. The study covers one repository, 19 tasks, and five attempts per task, leaving wide uncertainty. The audit was designed after the original experiments and is exploratory. No learned monitor was tested against restarts under matched conditions, so the work does not evaluate the detectors it is meant to inform.

The broader significance lies in the evaluation standard rather than any single result. The authors argue that developers should measure recovery opportunity, declare the intervention and its budget, and report uncertainty at the level of individual decisions. Whether monitors built to that standard can improve agent performance remains an open question, which the authors say requires further prospective testing.