No-Human in the Loop: Agentic Evaluation at Scale for Recommendation

TL;DR

ScalingEval is a comprehensive benchmarking framework that evaluates large language models as judges for recommendation tasks, providing reproducible, consensus-driven comparisons across multiple models and categories.

Contribution

The paper introduces ScalingEval, a scalable, multi-agent evaluation protocol for systematically comparing LLMs as judges without human annotation.

Findings

Claude 3.5 Sonnet achieves highest confidence

Gemini 1.5 Pro performs best overall

GPT-4o offers optimal latency-accuracy-cost balance

Abstract

Evaluating large language models (LLMs) as judges is increasingly critical for building scalable and trustworthy evaluation pipelines. We present ScalingEval, a large-scale benchmarking study that systematically compares 36 LLMs, including GPT, Gemini, Claude, and Llama, across multiple product categories using a consensus-driven evaluation protocol. Our multi-agent framework aggregates pattern audits and issue codes into ground-truth labels via scalable majority voting, enabling reproducible comparison of LLM evaluators without human annotation. Applied to large-scale complementary-item recommendation, the benchmark reports four key findings: (i) Anthropic Claude 3.5 Sonnet achieves the highest decision confidence; (ii) Gemini 1.5 Pro offers the best overall performance across categories; (iii) GPT-4o provides the most favorable latency-accuracy-cost tradeoff; and (iv) GPT-OSS 20B leads among open-source models. Category-level analysis shows strong consensus in structured domains (Electronics, Sports) but persistent disagreement in lifestyle categories (Clothing, Food). These results establish ScalingEval as a reproducible benchmark and evaluation protocol for LLMs as judges, with actionable guidance on scaling, reliability, and model family tradeoffs.