AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Giving AI agents a requirements-engineering phase before coding improves from-scratch programs

AI coding agents can patch bugs and add features inside large codebases, but building a program from nothing remains a stubborn challenge. On the ProgramBench benchmark, agents receive only natural-language documentation and an execute-only binary—a runnable program whose source code is hidden—as a behavioral oracle. Even the strongest model tested, GPT-5.5-high, fully re-implemented fewer than 1% of the 200 instances. A new study argues that the root problem is not coding skill but missing requirements engineering.

The researchers propose SpecFirst, a two-stage framework that splits the work into separate phases. First, a dedicated “spec agent” probes the binary through its normal command-line interface, observing standard output, standard error, and exit codes, then writes a structured specification file with headings for overview, flags, input, output format, error patterns, and edge cases. Only after that artifact exists does a code-synthesis agent begin implementing, using the specification as a stable reference. The authors argue this prevents two failure modes: behavioral intent drifting as long conversations grow, and early misinterpretations propagating uncorrected through the final code.

The paper illustrates the drift problem with the template renderer gomplate. In a baseline run using GPT-5.5-high, the agent discovered the full list of function namespaces at turn 4, including aws, gcp, sockaddr, semver, sprig, and cidr. Across the remaining 25 turns, none of those namespaces appeared again in its reasoning or tool calls. The final submission silently omitted all six, causing 126 hidden tests to fail before any code was written for them.

Across all 200 ProgramBench instances and four models—Qwen3.5-397B-A17B, Qwen3.6-35B-A3B, GPT-5.5-high, and GPT-5.4-mini—SpecFirst consistently beat the single-loop baseline, which used the same mini-SWE-agent scaffold without the separate elicitation phase. Average test pass rates rose by 6.9% to 21.3% relative to baseline. For example, Qwen3.5-397B went from 33.66% to 40.84% passed tests, and GPT-5.5-high rose from 59.02% to 65.14%. Binary exploration coverage also increased by 9.4% to 18.5% relative gains. The improvements were statistically significant in all cases. The strongest model also produced many more near-complete programs: the share of instances passing at least 95% of hidden tests quadrupled, from 1.5% to 6.5%.

The approach has clear tradeoffs. Mean per-instance API cost rose 48% to 130% across models; GPT-5.5-high, the most expensive, went from $2.54 to $5.85 per instance because the spec agent adds a full inference phase. And a good specification does not guarantee correct code. In a manual review of 50 failures, the researchers found that 52% came from execution-stage errors: the specification was correct and complete, but the implementation still diverged from it. That suggests stronger coding-time reasoning remains necessary.

The study does not claim to solve from-scratch program synthesis. Even with the best configuration, average pass rates leave substantial room for improvement. But the results make a broader point: treating behavioral specification elicitation as an explicit, first-class phase, the way human software engineers have long treated requirements gathering, may be a valuable design principle for AI agents tackling open-ended programming tasks.