AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluation

TL;DR

AJ-Bench is a comprehensive benchmark for evaluating agent-based judges in environment-aware verification tasks across multiple domains, showing performance improvements over traditional LLM-based judges.

Contribution

The paper introduces AJ-Bench, a new benchmark for systematically assessing agent-as-a-Judge in diverse complex environments, highlighting its capabilities and challenges.

Findings

Agent-as-a-Judge outperforms LLM-as-a-Judge baselines.

AJ-Bench covers 155 tasks across three domains.

Substantial open challenges remain in agent-based verification.

Abstract

As reinforcement learning continues to scale the training of large language model-based agents, reliably verifying agent behaviors in complex environments has become increasingly challenging. Existing approaches rely on rule-based verifiers or LLM-as-a-Judge models, which struggle to generalize beyond narrow domains. Agent-as-a-Judge addresses this limitation by actively interacting with environments and tools to acquire verifiable evidence, yet its capabilities remain underexplored. We introduce a benchmark AJ-Bench to systematically evaluate Agent-as-a-Judge across three domains-search, data systems, and graphical user interfaces-comprising 155 tasks and 516 annotated trajectories. The benchmark comprehensively assesses judge agents' abilities in information acquisition, state verification, and process verification. Experiments demonstrate consistent performance gains over LLM-as-a-Judge baselines, while also revealing substantial open challenges in agent-based verification. Our data and code are available at https://aj-bench.github.io/.