AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Specification-First AI Refactor Moves Across 189 Files Without Human Review

Reviewing AI-generated code is becoming a bottleneck in software engineering. A single large change can span hundreds of interdependent files, and the paper argues that at that scale, no human reviewer can hold the full dependency graph in mind. A new case study tests an alternative: audit the specification before code is generated, then hold the code to that frozen specification.

Joël Abenhaïm of AI Sovereign Labs reports the result in a manuscript posted July 31, 2026. The target was a 717,725-line TypeScript codebase, and the task was deliberately hard: dismantle a core architectural invariant in a streaming subsystem. Previously, closing a UI panel killed the underlying AI request. The new behavior required the generation to survive the panel closing, remain visible in a sidebar, and reattach to the same live stream on reopening without token loss or duplication. No pre-existing test encoded that behavior because the behavior did not exist before the change.

The agent, AICode running OpenAI’s ChatGPT 5.6 Sol model in max reasoning mode, followed a five-phase protocol: ideate, specify, refine, code, verify. First it wrote a formal specification from a natural-language request. Then 14 refinement cycles re-audited that specification against the real source, producing roughly 85 corrections and expanding planned scope from 110 to 160 files. After the specification was frozen, the agent generated code against it. Seventeen verification cycles then audited the code against the specification, correcting 116 deviations. In total, 201 defects were corrected before any human ran the program. Verification stopped only after two consecutive cycles returned zero findings.

One example illustrates the process. When asked to implement the frozen specification, the agent refused to start with a partial patch, warning that a partial implementation across 160 interdependent files, including 31 new ones, would leave the repository in an inconsistent state and violate the specification’s atomicity requirement. It proposed a ten-step decomposition; the operator instead asked for three larger segments, which the agent accepted.

The final refactor changed 189 files, 31 of them new. Combined with an earlier extraction commit, the operation touched 288 files, with 34,770 insertions and 16,422 deletions. It took three days and cost USD 2,430 in inference. On first manual execution, the specified behavior was present, the project’s pre-existing unit tests showed no regression, and roughly thirty later sessions produced no observed bug. The build was released as version 2.3.0.

The limitations are substantial, and the author states them plainly. This is one task, one codebase, one operator, and no control condition. The author designed the agent and performed the operation, so the report is self-reported even though 1,500+ pages of logs are published for inspection. The codebase is closed source, the result was obtained with one frontier model, and “no bug observed” is not proof of the absence of latent defects.

The broader significance is not that AI coding agents can now work without human oversight. It is that the point of control might be moved earlier in the process. Independent replication on a public codebase would be needed to establish whether specification-first convergence generalizes or was a carefully instrumented outlier.