AI Papers Reader

Personalized digests of latest AI research

View on GitHub

A Moving Target for Coding Benchmarks: SchrödingerRepo Tests Whether Agents Really Reason About Repositories

Coding agents are increasingly evaluated on SWE-bench, a benchmark of real GitHub issues with executable tests. But SWE-bench draws from popular open-source repositories that appear repeatedly in training data. That creates a measurement problem: a high score may mean an agent actually navigated and fixed a bug, or it may mean the model recognized familiar file names, APIs, and layouts. Researchers at Shanghai Jiao Tong University, Xi’an Jiaotong University, and East China Normal University propose SchrödingerRepo to separate those possibilities.

Instead of evaluating every agent against one fixed repository snapshot, SchrödingerRepo treats the repository as an evaluation-time latent variable. A seed generates a deterministic but different view of the same task each run. The transformation is designed to preserve executable behavior and the tests that define correctness, while removing familiar surface cues. It has four levels: reconstructing the problem statement; remapping repository paths, modules, and internal symbols into a virtual namespace; reordering definitions within files; and rewriting issue-relevant code into behaviorally equivalent variants. The mapping is invertible, so tool execution still runs against the real SWE-bench environment and patches are translated back for scoring. For the code-level changes, transformed repositories are validated to pass existing tests while leaving the target bug unfixed.

The paper’s case study shows what this changes. For django__django-11999, the original view let an agent search for get_.*_display under django/db/models, quickly locate the relevant files, and identify that Django installs a generated display method unconditionally. Under SchrödingerRepo, the same issue appears with transformed names and paths. The agent first explores the repository, searches display-related methods, opens transformed files such as storage_engine/object_models/entries/init.py and anchor.py, and checks tests and configuration. It still submits the same patch, but its actions increase from 37 to 217.

On SWE-bench Verified, enabling all four levels lowered Pass@1—the share of issues solved in a single run—by 6.0–14.4 percentage points across four model backends: GPT-5.4-mini fell from 46.8% to 35.6%, GPT 5.1 from 44.6% to 36.2%, DeepSeek-v4-Flash from 72.8% to 66.8%, and Gemini-3.1-Flash-Lite from 56.7% to 42.3%. Namespace mapping caused the largest individual drop. The extra work was mostly exploration: 81.6–83.6% of additional actions went to navigation, search, reading, and probing, with sharp increases in token use. On the repository-level question-answering benchmark SWE-QA, transformed views reduced average score by as much as 4.64 points and increased actions by 18.15–43.02%. On a March 2026 SWE-rebench split of 110 instances created after GPT-5.4-mini’s release, Pass@1 stayed at 17.27%, while actions rose 8.15% and input tokens 22.01%.

The authors interpret these results as evidence that current agents rely partly on familiar repository-side cues, not only on robust repository reasoning. But the study’s scope is limited: it focuses mainly on Python, open-source benchmarks, and command-line interaction. The researchers note that richer IDE or language-server tools may require adaptation, and they restricted two transformation levels to golden-patch-related files and regions to control cost. The work does not prove that SWE-bench scores are pure memorization. It does suggest that repository representation should become an explicit experimental variable, so evaluations measure robustness across semantically equivalent codebases rather than familiarity with one canonical snapshot.