Automatically Benchmarking LLM Code Agents through Agent-Driven Annotation and Evaluation

TL;DR

This paper introduces PRDBench, a diverse, project-level Python code benchmark constructed via agent-driven annotation, and a specialized, fine-tuned evaluation model achieving high human alignment for assessing code agents.

Contribution

It presents a novel agent-driven pipeline for creating diverse benchmarks and a dedicated, fine-tuned evaluation model to improve assessment accuracy of code agents.

Findings

PRDBench includes 50 real-world Python projects across 20 domains.

The specialized PRDJudge achieves over 90% human alignment.

The framework offers scalable, robust, and accurate evaluation of code agents.

Abstract

Recent advances in code agents have enabled automated software development at the project level, supported by large language models (LLMs). However, existing benchmarks for code agent evaluation face two major limitations. First, creating high-quality project-level evaluation datasets requires extensive domain expertise, leading to prohibitive annotation costs and limited diversity. Second, while recent Agent-as-a-Judge paradigms address the rigidity of traditional unit tests by enabling flexible metrics, their reliance on In-Context Learning (ICL) with general LLMs often results in inaccurate assessments that misalign with human standards. To address these challenges, we propose an agent-driven benchmark construction pipeline that leverages human supervision to efficiently generate diverse project-level tasks. Based on this, we introduce PRDBench, comprising 50 real-world Python projects across 20 domains, each with structured Product Requirement Documents (PRDs) and comprehensive criteria. Furthermore, to overcome the inaccuracy of general LLM judges, we propose a highly reliable evaluation framework powered by a specialized, fine-tuned model. Based on Qwen3-Coder-30B, our dedicated PRDJudge achieves over 90% human alignment in fixed-interface scenarios. Extensive experiments demonstrate that our suite provides a scalable, robust, and highly accurate framework for assessing state-of-the-art code agents.