Assistant-Guided Mitigation of Teacher Preference Bias in LLM-as-a-Judge

TL;DR

This paper introduces AGDe-Judge, a three-stage framework that uses an assistant model to mitigate teacher preference bias in LLM-based evaluation, improving fairness without sacrificing performance.

Contribution

We propose a novel three-stage framework incorporating an unbiased assistant model to effectively reduce teacher bias in LLM evaluation models.

Findings

AGDe-Judge reduces teacher preference bias significantly.

Maintains strong evaluation performance across six benchmarks.

Demonstrates effectiveness of assistant models in debiasing.

Abstract

LLM-as-a-Judge employs large language models (LLMs), such as GPT-4, to evaluate the quality of LLM-generated responses, gaining popularity for its cost-effectiveness and strong alignment with human evaluations. However, training proxy judge models using evaluation data generated by powerful teacher models introduces a critical yet previously overlooked issue: teacher preference bias, where the proxy judge model learns a biased preference for responses from the teacher model. To tackle this problem, we propose a novel setting that incorporates an additional assistant model, which is not biased toward the teacher model's responses, to complement the training data. Building on this setup, we introduce AGDe-Judge, a three-stage framework designed to debias from both the labels and feedbacks in the training data. Extensive experiments demonstrate that AGDe-Judge effectively reduces teacher preference bias while maintaining strong performance across six evaluation benchmarks. Code is available at https://github.com/Liuz233/AGDe-Judge.