AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Distilling the Full Software Life Cycle Helps a Small AI Build Programs From Scratch

Coding agents have grown adept at patching bugs, but building a program from nothing is much harder. On ProgramBench, a benchmark of 200 real command-line tools, even frontier models fully reproduce fewer than 1% of tasks; they often pass only part of the hidden tests. A new pipeline, MindForge, aims to teach smaller models the whole process by training them on an expert agent’s complete step-by-step records.

Researchers at Huawei Canada, Queen’s University, and University of Manitoba built MindForge to create training data without exposing source code. The pipeline compiles an open-source command-line tool, removes the source, and leaves an agent alone with the compiled executable and sanitized documentation. A teacher model, GLM-5.2, then probes the binary, designs an architecture, implements, debugs, tests, and refines until it produces a passing build. From 562 programs in six compiled languages, the team collected 1,001 such trajectories, repaired ones interrupted by infrastructure failures, and used refined versions to fine-tune Qwen3.6-27B, a 27-billion-parameter model.

The resulting model behaved differently when its commands failed. On a ProgramBench task involving Dropbear, the base Qwen3.6-27B noticed its usage output differed from the reference and stated, “The print_usage function needs to be fixed,” but then re-ran the same help command repeatedly, leaving the fix unapplied for roughly 30 actions. In another task, MindForge-27B, after seeing a help-format mismatch, edited its source code on the very next action. Across all 200 tasks, the fine-tuned model followed a failed command with an implementation edit 48.8% of the time, versus 31.8% for the base model.

These shifts came with large score gains. On ProgramBench’s average test pass rate, MindForge-27B rose from 37.98% to 49.51%—an 11.53 percentage-point gain, or about 30.4% relative improvement. That placed it above DeepSeek V4 Pro (47.80%) and near much larger models including GLM-5.1 (50.90%) and Opus 4.7 (51.38%), though still behind teacher GLM-5.2 (64.60%). It also improved on all seven benchmarks not used in training. The largest absolute gain was on repository translation RepoZero C2Rust, from 47.00% to 78.00% (+31.00 percentage points); DeepSWE rose from 1.76% to 15.92% (+14.16 points, roughly a ninefold relative increase).

The gains came with a real cost. Evaluating MindForge-27B on ProgramBench consumed 11.64 billion tokens across the 200 instances, about 5.7 times the base model’s 2.03 billion, because its trajectories were roughly twice as long. The authors note such long-horizon benchmarks are expensive, and several were run only once. They also caution that their rule-based analysis of which development activities mattered cannot, by itself, prove what caused the improvements.

The results do not “solve” from-scratch programming. But they suggest that whole-life-cycle supervision—specification discovery, design, debugging, and refinement together—can transfer long-horizon skill into a small model, and that the full messy job of building software is teachable.