Agent-as-a-Judge: Evaluate Agents with Agents

TL;DR

The paper introduces the Agent-as-a-Judge framework, enabling agentic systems to evaluate each other for more reliable and scalable self-improvement, demonstrated through a new code generation benchmark called DevAI.

Contribution

It presents the novel Agent-as-a-Judge framework that incorporates agentic features for evaluation, extending previous LLM-based approaches, and introduces DevAI benchmark for testing.

Findings

Agent-as-a-Judge outperforms LLM-as-a-Judge in evaluations.

It is as reliable as human evaluation baseline.

Demonstrates effectiveness on 55 AI development tasks.

Abstract

Contemporary evaluation techniques are inadequate for agentic systems. These approaches either focus exclusively on final outcomes -- ignoring the step-by-step nature of agentic systems, or require excessive manual labour. To address this, we introduce the Agent-as-a-Judge framework, wherein agentic systems are used to evaluate agentic systems. This is an organic extension of the LLM-as-a-Judge framework, incorporating agentic features that enable intermediate feedback for the entire task-solving process. We apply the Agent-as-a-Judge to the task of code generation. To overcome issues with existing benchmarks and provide a proof-of-concept testbed for Agent-as-a-Judge, we present DevAI, a new benchmark of 55 realistic automated AI development tasks. It includes rich manual annotations, like a total of 365 hierarchical user requirements. We benchmark three of the popular agentic systems using Agent-as-a-Judge and find it dramatically outperforms LLM-as-a-Judge and is as reliable as our human evaluation baseline. Altogether, we believe that Agent-as-a-Judge marks a concrete step forward for modern agentic systems -- by providing rich and reliable reward signals necessary for dynamic and scalable self-improvement.