Beyond Single-Point Judgment: Distribution Alignment for LLM-as-a-Judge

TL;DR

This paper introduces a distributional alignment framework for LLMs acting as evaluators, improving their reliability by matching judgment distributions to human data and enhancing robustness against limited annotations.

Contribution

It presents a novel distributional alignment training method using KL divergence and adversarial training to better reflect human judgment diversity in LLM evaluations.

Findings

Outperforms existing methods in alignment quality

Improves evaluation accuracy across tasks

Enhances robustness to limited data

Abstract

LLMs have emerged as powerful evaluators in the LLM-as-a-Judge paradigm, offering significant efficiency and flexibility compared to human judgments. However, previous methods primarily rely on single-point evaluations, overlooking the inherent diversity and uncertainty in human evaluations. This approach leads to information loss and decreases the reliability of evaluations. To address this limitation, we propose a novel training framework that explicitly aligns the LLM-generated judgment distribution with empirical human distributions. Specifically, we propose a distributional alignment objective based on KL divergence, combined with an auxiliary cross-entropy regularization to stabilize the training process. Furthermore, considering that empirical distributions may derive from limited human annotations, we incorporate adversarial training to enhance model robustness against distribution perturbations. Extensive experiments across various LLM backbones and evaluation tasks demonstrate that our framework significantly outperforms existing closed-source LLMs and conventional single-point alignment methods, with improved alignment quality, evaluation accuracy, and robustness.