AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Letting AI Agents Rewrite Their Own Scaffolding, With Guardrails

AI agents, which pair a language model with prompts, memory, tools, and workflow logic, often stop learning once they are deployed, even as the tasks they face change. A new paper from researchers at Zhongguancun Academy and the University of Science and Technology of China examines a way to address this: letting an agent’s surrounding software improve itself from its own experience while the underlying model stays fixed.

The authors call this “harness self-evolution.” A harness is the software and configuration that governs a model’s execution in a task environment. In each round, the current harness runs a batch of tasks, and the results are used to diagnose problems and propose revisions. Each candidate is assessed, and only admitted revisions persist into later runs. If nothing passes, the existing harness stays in place. The authors distinguish two kinds of change. Interpreted-state updates revise things like instructions, stored experience, and task routing. Executable revisions alter code, tool implementations, or runtime control logic, which is a larger change with a wider blast radius.

The paper offers a concrete illustration of the trade-offs. In a simulated office workload described in a cited report, a Codex-based agent generated spreadsheet tools and retained them for later tasks. On the first task, tool calls fell from 13 to 3, but wall-clock time rose 54.9% and token use rose 15.7%. Across five tasks, the cumulative figures showed savings of 110,351 tokens and 22.97 seconds. The paper notes, however, that the first-task totals and the cumulative series disagree on the first task’s cost, so the true cost trajectory remains unresolved.

On SpreadsheetBench, a spreadsheet-manipulation benchmark, a runtime-revision method called HarnessOpt reached 77.58% accuracy with GPT-5.4-nano when given skills, slightly above GPT-5.5 with the earlier SkillOpt method at 76.20%. Across four models, HarnessOpt with skills improved on SkillOpt-Lite by 5.69 to 11.39 percentage points. The authors caution that this comparison does not establish superiority at matched cost.

The paper’s theoretical contribution concerns when accepted revisions can be trusted. The authors propose admitting a candidate only when a high-probability upper bound on its risk change is negative, computed on a fresh set of validation tasks. Revisions informed by reused search data lack this independence, and the authors argue that such checks can mislead. The guarantees also rest on assumptions: they certify performance on the historical task mixture rather than on future distributions, and they require a bound on any gap between the evaluator’s scores and real-world performance.

The authors are also candid about risk. Persistent revisions can spread errors across tasks and teams, and software rollback cannot reverse physical harm from a faulty robot action or repair wrongly denied entitlements. They recommend keeping emergency stops, safety limits, and authorization checks outside the editable system.

The work is a framework and survey rather than a single new benchmark, and most of its empirical cases come from prior studies. Its main value may be in defining the problem clearly: once agents can rewrite their own scaffolding, validation, versioning, and independent safeguards become essential parts of the design.