AI Code Reviewers Get Granular: ‘Agentic Rubrics’ Boost Software Agent Verification
Verifying the quality of patches produced by Software Engineering (SWE) agents—Large Language Models (LLMs) trained to fix bugs and modify code—is a critical bottleneck in AI development. While traditional verification relies on executing code against unit tests, this process is expensive, slow, and often yields sparse pass/fail signals that fail to capture nuanced errors.
New research from Scale AI introduces Agentic Rubrics, a novel execution-free verification system that significantly improves the selection of correct AI-generated code fixes. Instead of relying on costly test execution, this system leverages an expert AI to generate a detailed, context-grounded scoring rubric by actively exploring the target codebase.
The approach is split into two phases. In the Rubric Generation Phase, an expert “rubric agent” interacts with a sandboxed repository—using tools like file search and inspection—to gather context about relevant code paths and project conventions. This allows the agent to synthesize a structured YAML checklist of criteria.
These rubrics are organized along four axes: File Change (minimal and local edits), Spec Alignment (satisfying issue requirements), Integrity (code hygiene and avoiding “cheating”), and Runtime (ensuring intended execution behavior).
In the Inference Phase, candidate patches proposed by the SWE agent are then scored against this bespoke rubric by an LLM judge, assigning a quantitative score between 0 and 1 without requiring any code execution.
This methodology proved highly effective when evaluated under parallel Test-Time Scaling (TTS) on the industry benchmark SWE-Bench Verified. Agentic Rubrics achieved a resolution rate of 54.2% for fixes generated by the Qwen3-Coder-30B-A3B model, marking a substantial gain of at least 3.5 percentage points over the strongest execution-free and test-based baselines.
The core strength of Agentic Rubrics lies in their ability to provide a dense, granular signal that moves beyond simple pass/fail outcomes. Researchers demonstrated that rubric scores align well with ground-truth tests but crucially identify subtle issues that existing tests miss.
For instance, in one example (matplotlib-26291), a candidate patch fixed a reported crash, allowing it to pass the ground-truth unit tests. However, the patch achieved this fix by inserting a dummy zero-size bounding box, which violated implicit contract rules within the Matplotlib library concerning how insets should be handled in “tight” figure layouts. The Agentic Rubric, because it was context-grounded, flagged this violation under the Integrity and Spec Alignment axes, resulting in a low score and effectively filtering out the incomplete fix.
This level of detail offers diagnostic feedback, pinpointing flaws like “unnecessary edits,” “missing edge cases,” or “wrong layer implementation”—issues that traditional tests often ignore. Furthermore, the cost analysis confirms efficiency: generating and grading rubrics is significantly cheaper than running other agentic verification methods, making it a scalable solution for future LLM training pipelines.
The findings suggest that distilling expert knowledge into structured, verifiable criteria allows AI agents to grade code quality holistically, ensuring that fixes are not only functionally correct but also maintain codebase integrity and follow project conventions.
Chat about this paper
To chat about this paper, you'll need a free Gemini API key from Google AI Studio.
Your API key will be stored securely in your browser's local storage.