CalibraEval: Calibrating Prediction Distribution to Mitigate Selection Bias in LLMs-as-Judges

TL;DR

CalibraEval is a novel, label-free method that reduces selection bias in LLM-based evaluations by adjusting prediction distributions through an optimization process, enhancing fairness and reliability.

Contribution

It introduces CalibraEval, a new approach that reformulates debiasing as an optimization problem and employs a non-parametric algorithm to mitigate bias without requiring labels.

Findings

Effectively reduces selection bias in LLM evaluations

Improves evaluation accuracy over existing methods

Demonstrates robustness across multiple benchmarks

Abstract

The use of large language models (LLMs) as automated evaluation tools to assess the quality of generated natural language, known as LLMs-as-Judges, has demonstrated promising capabilities and is rapidly gaining widespread attention. However, when applied to pairwise comparisons of candidate responses, LLM-based evaluators often exhibit selection bias. Specifically, their judgments may become inconsistent when the option positions or ID tokens are swapped, compromising the effectiveness and fairness of the evaluation result. To address this challenge, we introduce CalibraEval, a novel label-free method for mitigating selection bias during inference. Specifically, CalibraEval reformulates debiasing as an optimization task aimed at adjusting observed prediction distributions to align with unbiased prediction distributions. To solve this optimization problem, we propose a non-parametric order-preserving algorithm (NOA). This algorithm leverages the partial order relationships between model prediction distributions, thereby eliminating the need for explicit labels and precise mathematical function modeling.Empirical evaluations of LLMs in multiple representative benchmarks demonstrate that CalibraEval effectively mitigates selection bias and improves performance compared to existing debiasing methods. This work marks a step toward building more robust and unbiased automated evaluation frameworks, paving the way for improved reliability in AI-driven assessments