VRM: Teaching Reward Models to Understand Authentic Human Preferences

TL;DR

This paper introduces VRM, a novel variational reward modeling framework that better captures authentic human preferences by modeling the complex evaluation process, leading to improved alignment of language models with human judgments.

Contribution

VRM explicitly models human preference evaluation using latent variables for high- and low-dimensional features, offering a theoretical advantage and improved performance over existing reward models.

Findings

VRM outperforms existing reward models on benchmark datasets.

VRM provides a tighter generalization error bound.

Experimental results show VRM better captures authentic human preferences.

Abstract

Large Language Models (LLMs) have achieved remarkable success across diverse natural language tasks, yet the reward models employed for aligning LLMs often encounter challenges of reward hacking, where the approaches predominantly rely on directly mapping prompt-response pairs to scalar scores, which may inadvertently capture spurious correlations rather than authentic human preferences. In contrast, human evaluation employs a sophisticated process that initially weighs the relative importance of multiple high-dimensional objectives according to the prompt context, subsequently evaluating response quality through low-dimensional semantic features such as logical coherence and contextual appropriateness. Motivated by this consideration, we propose VRM, i.e., Variational Reward Modeling, a novel framework that explicitly models the evaluation process of human preference judgments by incorporating both high-dimensional objective weights and low-dimensional semantic features as latent variables, which are inferred through variational inference techniques. Additionally, we provide a theoretical analysis showing that VRM can achieve a tighter generalization error bound compared to the traditional reward model. Extensive experiments on benchmark datasets demonstrate that VRM significantly outperforms existing methods in capturing authentic human preferences.