An Empirical Study of Automating Agent Evaluation

TL;DR

This paper introduces EvalAgent, an AI system that automates complex agent evaluation by encoding domain expertise, and demonstrates its effectiveness through a new benchmark and meta-evaluation framework.

Contribution

The paper presents EvalAgent, a novel AI assistant for automated agent evaluation, and introduces AgentEvalBench and Eval@1 for systematic assessment.

Findings

EvalAgent improves Eval@1 from 17.5% to 65%.

EvalAgent achieves 79.5% human expert preference.

Removing evaluation skills drops Eval@1 from 65% to 30%.

Abstract

Agent evaluation requires assessing complex multi-step behaviors involving tool use and intermediate reasoning, making it costly and expertise-intensive. A natural question arises: can frontier coding assistants reliably automate this evaluation process? Our study shows that simply prompting coding assistants is insufficient for this task. Without domain-specific evaluation knowledge, frontier coding assistants achieve only a 30% execution success rate and produce over-engineered evaluations averaging 12+ metrics per agent, indicating that strong coding ability does not automatically translate to reliable agent evaluation. We introduce EvalAgent, an AI assistant that automates the end-to-end agent evaluation pipeline. EvalAgent encodes evaluation domain expertise as evaluation skills (procedural instructions, reusable code and templates, and dynamically retrieved API documentation) that compose into a trace-based pipeline producing complete evaluation artifacts including metrics, executable code, and reports. To systematically assess generated evaluations, we introduce a meta-evaluation framework alongside AgentEvalBench, a benchmark comprising 20 agents, each paired with evaluation requirements and test scenarios. We further propose the Eval@1 metric to measure whether generated evaluation code both executes and yields meaningful results on the first run. Our experiments show that EvalAgent produces focused evaluations, improving Eval@1 from 17.5% to 65%, and achieving 79.5% human expert preference over baseline approaches. Further ablation studies show that evaluation skills are critical for handling complex evaluation: removing them causes Eval@1 to drop significantly from 65% to 30%.