Any Large Language Model Can Be a Reliable Judge: Debiasing with a Reasoning-based Bias Detector

TL;DR

This paper introduces a reasoning-based bias detector (RBD) that externally identifies biases in large language model evaluations and guides self-correction, significantly improving evaluation reliability across multiple bias types and model scales.

Contribution

The paper presents RBD, a novel external module for bias detection and correction in LLM evaluators, addressing limitations of existing methods and demonstrating scalability and effectiveness.

Findings

RBD improves evaluation accuracy by 18.5% on average.

RBD enhances consistency by 10.9%.

RBD outperforms prompting baselines and fine-tuned judges.

Abstract

LLM-as-a-Judge has emerged as a promising tool for automatically evaluating generated outputs, but its reliability is often undermined by potential biases in judgment. Existing efforts to mitigate these biases face key limitations: in-context learning-based methods fail to address rooted biases due to the evaluator's limited capacity for self-reflection, whereas fine-tuning is not applicable to all evaluator types, especially closed-source models. To address this challenge, we introduce the Reasoning-based Bias Detector (RBD), which is a plug-in module that identifies biased evaluations and generates structured reasoning to guide evaluator self-correction. Rather than modifying the evaluator itself, RBD operates externally and engages in an iterative process of bias detection and feedback-driven revision. To support its development, we design a complete pipeline consisting of biased dataset construction, supervision collection, distilled reasoning-based fine-tuning of RBD, and integration with LLM evaluators. We fine-tune four sizes of RBD models, ranging from 1.5B to 14B, and observe consistent performance improvements across all scales. Experimental results on 4 bias types--verbosity, position, bandwagon, and sentiment--evaluated using 8 LLM evaluators demonstrate RBD's strong effectiveness. For example, the RBD-8B model improves evaluation accuracy by an average of 18.5% and consistency by 10.9%, and surpasses prompting-based baselines and fine-tuned judges by 12.8% and 17.2%, respectively. These results highlight RBD's effectiveness and scalability. Additional experiments further demonstrate its strong generalization across biases and domains, as well as its efficiency.