Agent-as-a-Judge

TL;DR

The paper discusses the transition from large language model-based evaluation to agent-based evaluation systems that utilize planning, tools, and collaboration for more reliable and nuanced assessments, and provides a comprehensive survey of this emerging field.

Contribution

It introduces a unified framework and taxonomy for agent-as-a-Judge, organizing methodologies, applications, and challenges in this evolving evaluation paradigm.

Findings

Agent-based evaluation enhances robustness and verifiability.

Survey organizes methodologies and applications across domains.

Identifies key challenges and future research directions.

Abstract

LLM-as-a-Judge has revolutionized AI evaluation by leveraging large language models for scalable assessments. However, as evaluands become increasingly complex, specialized, and multi-step, the reliability of LLM-as-a-Judge has become constrained by inherent biases, shallow single-pass reasoning, and the inability to verify assessments against real-world observations. This has catalyzed the transition to Agent-as-a-Judge, where agentic judges employ planning, tool-augmented verification, multi-agent collaboration, and persistent memory to enable more robust, verifiable, and nuanced evaluations. Despite the rapid proliferation of agentic evaluation systems, the field lacks a unified framework to navigate this shifting landscape. To bridge this gap, we present the first comprehensive survey tracing this evolution. Specifically, we identify key dimensions that characterize this paradigm shift and establish a developmental taxonomy. We organize core methodologies and survey applications across general and professional domains. Furthermore, we analyze frontier challenges and identify promising research directions, ultimately providing a clear roadmap for the next generation of agentic evaluation.