On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimization

TL;DR

This paper investigates the generalization limitations of the implicit reward model derived from Direct Preference Optimization in reinforcement learning from human feedback, revealing its poorer out-of-distribution performance compared to explicit reward models.

Contribution

It provides empirical evidence that DPORM has limited generalization ability, especially under distribution shifts, supporting the use of explicit reward models in iterative DPO methods.

Findings

DPORM fits training data well but generalizes less effectively.

DPORM's accuracy drops by 3-7% in out-of-distribution settings.

Explicit reward models outperform implicit models under distribution shifts.

Abstract

Reinforcement Learning from Human Feedback (RLHF) is an effective approach for aligning language models to human preferences. Central to RLHF is learning a reward function for scoring human preferences. Two main approaches for learning a reward model are 1) training an EXplicit Reward Model (EXRM) as in RLHF, and 2) using an implicit reward learned from preference data through methods such as Direct Preference Optimization (DPO). Prior work has shown that the implicit reward model of DPO (denoted as DPORM) can approximate an EXRM in the limit. DPORM's effectiveness directly implies the optimality of the learned policy, and also has practical implication for LLM alignment methods including iterative DPO. However, it is unclear how well DPORM empirically matches the performance of EXRM. This work studies the accuracy at distinguishing preferred and rejected answers for both DPORM and EXRM. Our findings indicate that even though DPORM fits the training dataset comparably, it generalizes less effectively than EXRM, especially when the validation datasets contain distribution shifts. Across five out-of-distribution settings, DPORM has a mean drop in accuracy of 3% and a maximum drop of 7%. These findings highlight that DPORM has limited generalization ability and substantiates the integration of an explicit reward model in iterative DPO approaches.