AI Papers Reader

Personalized digests of latest AI research

View on GitHub

AI Research Loop Finds Cheaper Ways for Coding Agents to Work

As coding agents move from short, supervised completions to long unattended runs, every extra model call, log line, and repeated context chunk adds to token cost. A team at NVIDIA, NTU, and MIT attacked that problem at the harness layer—the software that mediates between a model and its tools, files, and execution feedback—rather than retraining the model or changing serving infrastructure.

Their system, SoL-Pi, treats harness improvement as an auto-research loop. A research AI observes execution traces from a separate agent running the base harness, proposes candidate changes, and tests them in prepared environments. Candidates must stay within fixed capability tolerances and improve an efficiency metric. Independent reviewers and development validation refine promising ideas; final held-out benchmarks are frozen and never feed back into search. The search spanned 152 proposed directions, 535 executable environments, more than 3,000 runs, and more than 60,000 agent–environment interactions.

Four mechanisms survived selection. Action Fusion combines a file mutation with a follow-up command; in the paper’s illustrated path, an edit plus a test command cuts three API calls to two by returning both outcomes in one observation. Online Context Compact decides when to compact based on plan-step completion and a cost gate comparing projected input savings with prompt-cache rewrite costs. ObservationPack archives tool outputs larger than 10 KiB and, after sending them in full for two provider requests, replaces them with a stable handle and a 1 KB excerpt. Evidence-Preserving Reducer compresses build and test logs of at least 4 KiB using GPT-5.6 Luna, then verifies the receipt with checks on schema, source hash, exit status, exact quotes, and size, falling back to the original log if verification fails.

On EdgeBench’s 51 tasks, the complete efficiency-oriented SoL-Pi stack scored 42.0 with GPT-5.6 Sol, compared with 44.8 for Pi—93.7% of Pi’s average score—while reducing recorded token traffic by 49.0% (1.10 billion vs. 2.15 billion tokens) and API cost by 33.2% ($894 vs. $1,339). On Claude Opus 5, without further search or adaptation, it scored 42.2 versus Pi’s 44.8, reducing token traffic by 44.7% and API cost by 33.5% ($1,158 vs. $1,741). A performance-oriented single mechanism, ObservationPack on GPT-5.6 Sol, raised average score from 44.8 to 47.2 (5.3%) while reducing token traffic by 6.1%.

The gains come with tradeoffs. The efficiency stack gives up some score, and on 63 CPU-only Terminal-Bench 4 tasks, SoL-Pi solved 15 tasks versus 18 for both Codex and Pi, although its total model cost was 26.3% lower than Pi’s and cost per solved task was 11.6% lower. The authors also note the harness was optimized on GPT-5.6 Sol trajectories; on Opus 5, its mechanisms triggered less often and less intensively, though they still improved token efficiency when they activated. They call the results preliminary and say recursive efficient improvement—using a cheaper harness to build an even cheaper successor—remains a long-term vision, not a demonstrated compounding effect.

The work suggests that automated searches over harness components can discover reusable efficiency mechanisms that transfer across models, pointing toward lower-cost long-horizon agents. But it establishes gains on specific benchmarks and settings, not a general solution to agent cost or self-improvement.