AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Harness Design for Coding Agents Is Conditional, Study Finds

Coding agents increasingly close GitHub issues and execute terminal tasks, but their performance comes from a software harness surrounding the model. Existing evaluations often compare entire harnesses, making it unclear whether gains come from planning, tool interfaces, context management, or interactions with the model. In a new empirical study, researchers built a lightweight coding harness with a fixed execution loop and varied three components: an explicit planning scaffold, an action space of predefined tools versus a bash-only shell, and five context-management policies. They evaluated 176 settings across Nemotron-3 30B, 120B, and 550B and Mistral-Medium-3.5-128B on SWE-Bench Verified and Terminal-Bench 2.1, at context-window budgets of 32k, 64k, 96k, and 128k tokens.

The policies were no compaction (T0), elision alone (T1), elision plus recall (T2), LLM summarization (T3), and staged elision before summarization (T4). The researchers report that context management matters most when the window is tight. Averaged over models, the success-rate gap between managed tiers T1–T4 and T0 fell from 35.7 to 2.7 percentage points on SWE-Bench and from 9.5 to 2.8 points on Terminal-Bench as the budget expanded from 32k to 128k. The mechanism they identify is overflow avoidance: T0’s window-overflow failure rate dropped from 78.7% to 8.7% on SWE-Bench and 61.0% to 12.1% on Terminal-Bench, while all managed tiers had zero overflow failures. T4 showed the strongest cost profile among managed strategies, with comparable success and the lowest mean cost at every budget; recall was rarely used, and T2 did not improve accuracy over T1 (mean difference −0.36 percentage points).

Planning’s role shifted with model capability. For Nemotron-3 30B, enabling planning at T4/128k raised SWE-Bench success by 11.6 percentage points and Terminal-Bench by 4.5 points, at higher cost. Trajectory analysis showed why: without planning, the median SWE-Bench run collapsed from 40 to 5 turns, 68.6% of runs ended without an edit, and 58.4% stalled at file localization; with planning, those figures were 27.8% and 10.4%. For stronger models, planning mainly saved cost. On SWE-Bench, Nemotron-3 550B and Mistral-Medium-3.5-128B cut costs by roughly 30% and 32%, respectively, while success fell 2.0 and 0.4 percentage points.

Action-space results were likewise conditional. Predefined tools raised success for Nemotron-3 30B by 15.0 percentage points on SWE-Bench and 10.1 points on Terminal-Bench. But bash-only improved Nemotron-3 550B by 3.6 points on SWE-Bench and 5.6 points on Terminal-Bench while reducing cost by 53% and 30%; Mistral preferred full tools on SWE-Bench but bash-only on the more shell-centric Terminal-Bench. The authors caution that planning and action-space ablations were run only under the T4/128k configuration, Terminal-Bench has just 89 tasks, and the action-space change bundles tool availability, prompts, state tracking, and diagnostics. They also note SWE-Bench Verified is Python-only and model size is an imperfect proxy for capability.

The study does not crown a universal harness. Instead, it argues that context management reduces sensitivity to window capacity, planning helps weak models start editing but trims redundant verification in strong ones, and tool interfaces should match a model’s shell proficiency and task. For builders, the practical message is that harness components should be selected for the model, task type, and budget rather than adopted by default.