CyclicJudge: Mitigating Judge Bias Efficiently in LLM-based Evaluation

TL;DR

CyclicJudge is a novel evaluation method that reduces judge bias in LLM assessments by using a round-robin judge assignment, improving reliability without increasing costs.

Contribution

The paper introduces CyclicJudge, a round-robin judge assignment strategy that optimally mitigates bias in LLM evaluation with fixed judge panels.

Findings

CyclicJudge recovers the panel mean score exactly.

Empirical validation on MT-Bench and MindEval shows effectiveness.

CyclicJudge matches single-judge evaluation cost while reducing bias.

Abstract

LLM-as-judge evaluation has become standard practice for open-ended model assessment; however, judges exhibit systematic biases that cannot be averaged out by increasing the number of scenarios or generations. These biases are often similar in magnitude to the model differences that benchmarks are designed to detect, resulting in unreliable rankings when single-judge evaluations are used. We introduce a variance decomposition that partitions benchmark score variance into scenario, generation, judge, and residual components. Based on this analysis, CyclicJudge, a round-robin assignment of judges to scenarios, is demonstrated to be the optimal strategy for a fixed judge panel and judge-call budget: the score recovers the panel mean exactly while matching the cost of single-judge evaluation. Empirical results on MT-Bench and MindEval validate the effectiveness of CyclicJudge as predicted, across both general-purpose and domain-specific evaluation settings.