AlphaEval: Evaluating Agents in Production

TL;DR

AlphaEval introduces a comprehensive, production-realistic benchmark of 94 tasks from seven companies, evaluating complete AI agents across diverse real-world scenarios and establishing a systematic methodology for creating such benchmarks.

Contribution

The paper presents a novel production-grounded benchmark and a systematic framework to transform real-world requirements into evaluation tasks efficiently.

Findings

Evaluates complete AI agents as commercial systems, revealing performance variations unseen in model-centric benchmarks.

Includes diverse evaluation paradigms such as LLM-as-a-Judge, formal verification, and UI testing.

Provides a reproducible methodology for organizations to create their own production-relevant benchmarks.

Abstract

The rapid deployment of AI agents in commercial settings has outpaced the development of evaluation methodologies that reflect production realities. Existing benchmarks measure agent capabilities through retrospectively curated tasks with well-specified requirements and deterministic metrics -- conditions that diverge fundamentally from production environments where requirements contain implicit constraints, inputs are heterogeneous multi-modal documents with information fragmented across sources, tasks demand undeclared domain expertise, outputs are long-horizon professional deliverables, and success is judged by domain experts whose standards evolve over time. We present AlphaEval, a production-grounded benchmark of 94 tasks sourced from seven companies deploying AI agents in their core business, spanning six O*NET (Occupational Information Network) domains. Unlike model-centric benchmarks, AlphaEval evaluates complete agent products -- Claude Code, Codex, etc. -- as commercial systems, capturing performance variations invisible to model-level evaluation. Our evaluation framework covers multiple paradigms (LLM-as-a-Judge, reference-driven metrics, formal verification, rubric-based assessment, automated UI testing, etc.), with individual domains composing multiple paradigms. Beyond the benchmark itself, we contribute a requirement-to-benchmark construction framework -- a systematic methodology that transforms authentic production requirements into executable evaluation tasks in minimal time. This framework standardizes the entire pipeline from requirement to evaluation, providing a reproducible, modular process that any organization can adopt to construct production-grounded benchmarks for their own domains.