Contrastive Decoding Mitigates Score Range Bias in LLM-as-a-Judge

TL;DR

This paper addresses the score range bias in large language models used as evaluators and proposes contrastive decoding to improve the correlation with human judgments in summarization tasks.

Contribution

It identifies the score range bias issue in LLM evaluators and introduces contrastive decoding as a method to mitigate this bias, enhancing evaluation reliability.

Findings

Contrastive decoding reduces score range bias in LLM evaluations.

Achieved up to 11.7% relative improvement in correlation with human judgments.

Bias exists among models from the same family, not just across different models.

Abstract

Large Language Models (LLMs) are commonly used as evaluators in various applications, but the reliability of the outcomes remains a challenge. One such challenge is using LLMs-as-judges for direct assessment, i.e., assigning scores from a specified range without any references. Focusing on summarization, we first show that this challenge stems from LLM judge outputs being associated with score range bias, i.e., LLM judge outputs are highly sensitive to pre-defined score ranges. We also show that similar biases exist among models from the same family. We then mitigate this bias through contrastive decoding, achieving up to 11.7% relative improvement on average in Spearman correlation with human judgments across different score ranges.