Beyond Scalar Reward Model: Learning Generative Judge from Preference Data

TL;DR

This paper introduces a generative judge trained from preference data using LLMs to produce rationales and judgments, improving interpretability and robustness over traditional scalar reward models.

Contribution

It proposes a novel method to train a generative judge with self-generated contrastive judgments, eliminating the need for a reward head and enhancing interpretability and bias robustness.

Findings

Performance comparable to scalar reward models on preference data

Superior interpretability due to natural language rationales

Greater robustness against dataset biases

Abstract

Learning from preference feedback is a common practice for aligning large language models~(LLMs) with human value. Conventionally, preference data is learned and encoded into a scalar reward model that connects a value head with an LLM to produce a scalar score as preference or reward. However, scalar models lack interpretability and are known to be susceptible to biases in datasets. This paper investigates leveraging the generation capability of LLMs to address both limitations in one shot. Specifically, we prompt the pre-trained LLM to generate positive and negative judgments, both supported with rationales in natural language form. The self-generated contrastive judgment pairs are used to train the generative judge with Direct Preference Optimization (DPO). This proposal of training the generative Judge using self-generated Contrastive judgments (Con-J) ensures natural interpretability due to the generated rationales together with the judgments, as well as high robustness against bias without the need for an additional reward head. Experimental results show that the performance of Con-J is comparable to the scalar reward model trained on the same collection of preference data, and demonstrate its superior interpretability and robustness in encoding human preferences.