Mediocrity is the key for LLM as a Judge Anchor Selection

TL;DR

This paper investigates how the choice of anchor models affects the reliability of LLM-based evaluations, revealing that poor anchor selection can significantly distort model rankings and proposing guidelines for better anchor choices.

Contribution

It systematically analyzes the impact of anchor selection in LLM evaluation, providing empirical evidence and actionable recommendations for more reliable benchmarking.

Findings

Anchor choice significantly affects evaluation correlation with human rankings.

Extreme anchors (best/worst models) are poor choices for reliable evaluation.

Standard benchmark sizes are often insufficient for distinguishing competitive models.

Abstract

The ``LLM-as-a-judge'' paradigm has become a standard method for evaluating open-ended generation. To address the quadratic scalability costs of pairwise comparisons, popular benchmarks like Arena-Hard and AlpacaEval compare all models against a single anchor. However, despite its widespread use, the impact of anchor selection on the reliability of the results remains largely unexplored. In this work, we systematically investigate the effect of anchor selection by evaluating 22 different anchors on the Arena-Hard-v2.0 dataset. We find that the choice of anchor is critical: a poor anchor can dramatically reduce correlation with human rankings. We identify that common anchor choices (best-performing and worst-performing models) make poor anchors. Because these extreme anchors are consistently better or worse than all other models, they are seldom indicative of the relative ranking of the models. We further quantify the effect size of anchor selection, showing it is comparable to the selection of a judge model. We conclude with actionable recommendations. First, we conduct a power analysis, and compute sufficient benchmark sizes for anchor-based evaluation, finding that standard benchmark sizes are insufficient for pairwise evaluation and fail to distinguish between competitive models reliably. Second, we provide guidelines for selecting informative anchors to ensure reliable and efficient evaluation practices.