AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Benchmark Asks Whether AI Agents Can Advance Real Computer Science Research

Can AI agents make genuine contributions to computer science research, or only solve self-contained programming puzzles? A new benchmark, FrontierCS 2, tries to answer that question by measuring agents against the code that researchers actually released with their papers.

Existing tests of AI coding ability typically give an agent a problem statement, have it write one program, and grade that program once. Research looks different. A researcher inherits a working system, changes one component, and is judged against the original implementation on the workloads the paper reported. The FrontierCS 2 team, drawn mainly from UC Berkeley, Princeton and other universities, built its benchmark around that setting.

The method works by subtraction. Researchers take a published paper and its authors’ public code, delete the code implementing the paper’s main contribution, and keep the deleted code as a “human reference,” the bar agents must clear. Each agent receives the repository without the contribution and may submit patches at any point. A separate judge scores each patch on a development suite of workloads, while keeping the evaluator, a held-out final suite and the reference private. When the run ends, the agent names one final patch, which the judge tests on the held-out suite. A run passes only if the patch beats the reference on every metric and every held-out workload. The benchmark contains 48 tasks across 13 domains, from machine-learning systems to databases and cryptography.

One case shows the kind of ingenuity involved. In QuarterPIR, a private information retrieval scheme that lets a client read one database entry without revealing which one, the human reference sends extra parity values with each reply. A passing run from Muse Wonder 1.0 replaced them with two checksums, the construction used in RAID-6 disk arrays. On a 67-million-entry database, this cut the bytes per lookup from 1,760 to 1,280, roughly 27 percent fewer. The authors note that they found this idea only by reading the patches, because neither the pass criterion nor the final score singled it out.

The headline result is that Claude Opus 5.5, run in Claude Code, passed 58 percent of its runs. GPT-6 Astra passed 44 percent, and every other agent passed fewer than half. Pass rates ranged down to 8 percent for Step 5 Preview. No agent passed 16 of the tasks. On the authors’ capability index, which places agents and tasks on a shared scale, Claude Opus 5.5 scored 55.7, and the median task sat near the strongest agent’s level.

The authors also report that a higher final score does not guarantee a pass. Most runs that beat the reference’s final score still missed a threshold on some metric or workload. On SVG-EAR, a sparse video attention task, GPT-6 Astra was faster than the reference on every clip and closer to full attention on average, yet it still failed on image quality in two clips. In the packet-scheduler task, the final score ranked first a design that used the most lookup tables among passing circuits, because the score rewards speed and ignores circuit size.

The study has clear limits. Each agent ran each task once, and the ranking of leading agents shifts with the cost budget per run. The authors recommend reporting such side costs alongside pass rates. Because the results rest on 48 tasks, single runs and one harness per model, they describe where current agents stand on this set rather than how close they are to performing research engineering in general. A private set of more than 100 tasks, to be announced later, will test whether the findings hold.