LRHP: Learning Representations for Human Preferences via Preference Pairs

TL;DR

This paper introduces LRHP, a novel framework for learning structured human preference representations from preference pairs, enabling better analysis and broader application beyond traditional reward modeling in RLHF.

Contribution

The work presents a new preference representation learning task and a generalizable framework that extends beyond reward modeling to improve downstream preference-related tasks.

Findings

Significantly outperforms baselines in preference data selection.

Achieves strong results in preference margin prediction.

Demonstrates utility of structured preference representations.

Abstract

To improve human-preference alignment training, current research has developed numerous preference datasets consisting of preference pairs labeled as "preferred" or "dispreferred". These preference pairs are typically used to encode human preferences into a single numerical value through reward modeling, which acts as a reward signal during reinforcement learning from human feedback (RLHF). However, representing these human preferences as a numerical value complicates the analysis of these preferences and restricts their broader applications other than RLHF. In contrast, in this work, we introduce a preference representation learning task that aims to construct a richer and more structured representation of human preferences. We further develop a more generalizable framework, Learning Representations for Human Preferences via preference pairs (namely LRHP), which extends beyond traditional reward modeling to tackle this task. We verify the utility of preference representations in two downstream tasks: preference data selection and preference margin prediction. Building upon the human preferences in representations, we achieve strong performance in both tasks, significantly outperforming baselines.