Agentic Rubrics as Contextual Verifiers for SWE Agents

TL;DR

This paper introduces Agentic Rubrics, a novel approach where an expert agent creates context-specific checklists to verify software patches without executing code, improving scalability and interpretability in software engineering agents.

Contribution

The paper presents Agentic Rubrics, a new method for code verification that leverages expert agents to generate context-grounded checklists, outperforming existing scalable verification techniques.

Findings

Achieved 54.2% score on Qwen3-Coder-30B-A3B with Agentic Rubrics.

Rubric scores align with ground-truth tests and identify issues tests miss.

Context gathering is crucial for effective, unambiguous verification.

Abstract

Verification is critical for improving agents: it provides the reward signal for Reinforcement Learning and enables inference-time gains through Test-Time Scaling (TTS). Despite its importance, verification in software engineering (SWE) agent settings often relies on code execution, which can be difficult to scale due to environment setup overhead. Scalable alternatives such as patch classifiers and heuristic methods exist, but they are less grounded in codebase context and harder to interpret. To this end, we explore Agentic Rubrics: an expert agent interacts with the repository to create a context-grounded rubric checklist, and candidate patches are then scored against it without requiring test execution. On SWE-Bench Verified under parallel TTS evaluation, Agentic Rubrics achieve a score of 54.2% on Qwen3-Coder-30B-A3B and 40.6% on Qwen3-32B, with at least a +3.5 percentage-point gain over the strongest baseline in our comparison set. We further analyze rubric behavior, showing that rubric scores are consistent with ground-truth tests while also flagging issues that tests do not capture. Our ablations show that agentic context gathering is essential for producing codebase-specific, unambiguous criteria. Together, these results suggest that Agentic Rubrics provide an efficient, scalable, and granular verification signal for SWE agents.