AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Reference-Guided Coding Benchmark Finds Agents Falter as Missing Features Compound

Coding agents are usually evaluated on issues, instructions, or tests that already describe desired behavior. In real web development, however, developers often infer behavior by using a working reference—an earlier version, a prototype, or a comparable app—and then implementing that behavior in an incomplete codebase. A new benchmark called ProgramDistill asks whether coding agents can do the same: discover what a live reference application does, recover it in an editable application whose source has been masked, and verify the result.

The benchmark is built with an automated pipeline the authors call mine–craft–patch. Mining explores 26 interactive web applications and records replayable browser interactions as verified behavior traces. Crafting turns those traces into repair tasks by deleting the source implementation behind a behavior. Patching then asks a coding agent to restore the missing behavior by interacting with the unmodified reference, editing the current application, and validating against recorded traces. The work produced 1,975 replay-verified behaviors and 4,063 tasks—2,862 atomic and 1,201 cumulative.

A key feature is restoration depth. Because web-app behaviors have prerequisites, such as logging in before creating a board, tasks can be composed along a prerequisite lineage. Atomic tasks require one repair; cumulative tasks require several dependent repairs to work together. The researchers evaluated nine frontier models on ProgramDistill-300, a stratified 300-task suite, using binary success and chain scores that give partial credit for recovered prefixes.

GPT-6 Astra achieved the highest mean binary score in partial-application reconstruction, 84.3%, followed by Claude Opus 5 at 68.7% and GPT-5.6 Sol at 60.7%. Their mean trajectory costs were $33.99, $28.86, and $16.01, respectively. Astra’s advantage over Opus 5 was 15.7 percentage points, at about 18% higher mean trajectory cost. Performance degraded with composition: Astra scored 100% at restoration depth 1 but 64.0% at depth 8, a 36-percentage-point drop. Opus 5 fell from 96% to 32%, and Sol from 92% to 32%.

The paper also documents a concrete failure mode. In one full-application reconstruction run, GPT-6 Astra rebuilt card dragging in Trello (Knowankit), but a center drop left cards in the opposite order from the reference. After its final relevant edit, the agent validated other board interactions rather than replaying that drag workflow, so the ordering error went undetected. More broadly, across 977 failed atomic behaviors in full-application reconstruction, 59.2% were never observed in the reference; 27.9% involved the wrong state, route, or result after observation.

Starting from a minimal scaffold, GPT-6 Astra recovered 49.15% of cumulative workflows under the binary score, compared with 28.81% for Opus 5 and 21.07% for Sol. Astra’s chain score was 96.2% on logic-only tasks and 84.9% on logic-and-UI tasks, an 11.3-point gap, indicating that reconstructing the interface remains harder even when partial restoration earns credit.

The authors note limitations: agents receive structured browser observations but not screenshots; the corpus consists of self-contained web applications; and the construction pipeline uses GPT-5.6 Sol, so task quality and cost may depend on that model. They present ProgramDistill as a diagnostic benchmark and a possible basis for future training, not as proof that agents can already match reference-guided development.