ImplicitRM: Unbiased Reward Modeling from Implicit Preference Data for LLM alignment

TL;DR

This paper introduces ImplicitRM, a novel method for learning unbiased reward models from implicit human feedback data, reducing costs and overcoming bias and data limitations in LLM alignment.

Contribution

ImplicitRM is the first approach to effectively learn reward models from implicit feedback by stratifying samples and deriving an unbiased likelihood-based training objective.

Findings

ImplicitRM achieves accurate reward modeling from implicit data.

The method effectively addresses bias and lack of negative samples.

Experimental results outperform existing approaches.

Abstract

Reward modeling represents a long-standing challenge in reinforcement learning from human feedback (RLHF) for aligning language models. Current reward modeling is heavily contingent upon experimental feedback data with high collection costs. In this work, we study \textit{implicit reward modeling} -- learning reward models from implicit human feedback (e.g., clicks and copies) -- as a cost-effective alternative. We identify two fundamental challenges in implicit reward modeling: (1) Implicit preference data lacks definitive negative samples, which makes standard positive-negative classification methods inapplicable; (2) Implicit preference data suffers from user preference bias, where different responses have different propensities to elicit user feedback actions, which exacerbates the difficulty of distinguishing definitive negative samples. To address these challenges, we propose ImplicitRM, which aims to learn unbiased reward models from implicit preference data. ImplicitRM stratifies training samples into four latent groups via a stratification model. Building on this, it derives a learning objective through likelihood maximization, which we prove is theoretically unbiased, effectively resolving both challenges. Experiments demonstrate that ImplicitRM learns accurate reward models across implicit preference datasets. Code is available on our project website.