AI Papers Reader

Personalized digests of latest AI research

View on GitHub

When “All Tests Pass” Masks a Failed Software Migration

Swapping the language, framework, or build system of a mature software project is a costly, mostly manual job. AI coding assistants that excel at fixing bugs have raised a question: can they take over these whole-repository migrations? A new benchmark, SWE Refactor Bench, suggests the answer is largely no—and exposes why ordinary test suites cannot tell a real migration from a fake one.

The problem is what the paper calls Blindness. Existing coding benchmarks measure behavior: run a fixed set of tests and see whether they pass. That works for bug fixes because the starting code fails. But a migration starts with a repository whose tests already pass, so an agent can return the code unchanged and earn a perfect score. Behavioral tests can show nothing was broken; they cannot show anything was changed.

To close that gap, the researchers built 20 migration tasks from real open-source infrastructure—SQLite, zlib, libsodium, and GraphHopper among them—spanning language, framework, platform, and build-toolchain rewrites. Each submission passes through three gates. Migration Audit uses a model judge to ask whether the old stack actually disappeared. Behavioral Tests run 130,118 fixed checks recorded from the original system, with every one required to pass. Agentic Verification then sends six independent coding agents, each with one hour, to hunt for hidden behavioral differences; a verifier can reject a submission only with an executable counterexample that works on the original and fails on the migrated version.

The audit catches a failure mode no behavioral test can express. On lang01—migrating the C Markdown parser cmark to Rust—five of 26 runs passed every fixed check, yet all five were blindness: the code had never really been migrated. In one claude-opus-5 run, all 4,184 behavioral checks passed and seven of eight audit criteria were met, but the eighth failed because the Rust was a line-by-line transliteration of the C, carrying the original’s manual memory management into unsafe Rust instead of redesigning ownership. Behavior was preserved; the migration’s purpose was not.

Across 520 runs involving eight frontier models and 26 model-effort configurations, only 28 runs (5.4 percent) passed all three stages, and 13 of the 20 tasks received no accepted solution at all. The strongest result was claude-opus-5 at high effort, scoring 47.0 out of 100; gpt-5.6-sol, kimi-k3, and claude-sonnet-5 trailed at 28.5, 19.5, and 15.0, respectively.

The results also separate two skills. Thirty runs preserved behavior by skipping the migration and were stopped at the audit; 252 completed the migration but broke behavior. Even among the 340 runs that did migrate, 58 percent reached 99 percent of the fixed checks but only 26 percent reached 100 percent. And of the 88 submissions that cleared both earlier gates, agentic verifiers found counterexamples in 60.

The authors caution that an accepted migration is not proven equivalent—it survived the strongest verification panel they could field, and scores would shift with stronger verifiers. That caveat matters: the benchmark is a stress test of current abilities, not a ranking of all migration difficulty. Still, by showing that “getting the code right” and “getting the migration done” are distinct abilities, it gives the field a way to measure progress on maintenance work that remains stubbornly human.