AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Before Writing Code, AI Software Agents Need to Probe the Unknown

When large language models (LLMs) are asked to fix bugs in an existing repository, they often perform impressively. But when tasked with rebuilding complex software entirely from scratch—guided only by surface-level documentation and an execute-only binary—even state-of-the-art models struggle, often fully resolving fewer than 1% of tasks.

The root cause isn’t necessarily a lack of coding intelligence, but a flaw in how AI agent frameworks operate. Standard frameworks mix reading documentation, testing the binary, and writing code into a single, unstructured loop. As the agent’s context window fills up over dozens of turns, it loses track of subtle behavioral details, misses edge cases, and lets early misinterpretations silently corrupt the final program.

To solve this, researchers from Huawei Canada, Queen’s University, University of Manitoba, and Concordia University have introduced SpecFirst, a framework inspired by traditional software engineering principles: explicitly figure out what to build before writing a single line of code.

Test First, Code Second

Instead of forcing a single AI model to juggle exploration and implementation simultaneously, SpecFirst splits the task into a two-stage pipeline:

  1. The Spec Agent: Acting as a requirements engineer, this agent systematically probes the target program’s executable binary via command-line interactions. It tests boundary conditions, triggers error paths, and evaluates combined flags to observe stdout, stderr, and exit codes. It then freezes these behavioral discoveries into a structured Markdown file (SPEC.md).
  2. The Code Synthesis Agent: Equipped with the original documentation, the binary, and the newly generated SPEC.md, a second agent focuses exclusively on implementation.

To understand why this matters, consider gomplate, a complex command-line template renderer featured in the benchmark. A standard AI agent usually reads the tool’s basic overview and jumps straight into coding. However, gomplate supports hundreds of built-in functions across diverse namespaces like math, net, and crypto that brief documentation leaves underspecified.

In experiments, standard agents often lost track of these namespaces mid-run or mangled typed function signatures—such as degrading a strict signature like Seq(start, end, step int64) into a loose, broken interface during multi-turn refactoring. Under SpecFirst, the Spec Agent systematically discovers these functions and locks their exact behavior into SPEC.md before coding starts, giving the developer agent an unshakeable reference point that won’t drift as context grows.

Superior Codebases and Higher Pass Rates

Evaluated across all 200 command-line software instances in the ProgramBench suite using four different LLMs (including GPT-5.5-high and Qwen models), SpecFirst achieved statistically significant gains across the board:

  • Higher Pass Rates: Average test pass rates rose by 6.9% to 21.3% across evaluated models compared to traditional single-loop baselines.
  • Expanded Probing Coverage: The Spec Agent expanded binary exploration coverage by 9.4% to 18.5%, uncovering logic that standard agents overlooked.
  • Near-Perfect Code: On the hardest tasks, GPT-5.5-high improved its average test score from 30.8% to 40.0%, while tripling the proportion of near-perfect software implementations (passing ≥90% of tests) from 5.5% to 16.5%.

Analysis of agent behavior showed that providing a structured specification upfront fundamentally alters how coding models budget their effort: rather than quitting early due to uncertainty, agents begin writing code sooner and construct larger, more complete software systems.