Robust Preference Optimization through Reward Model Distillation

TL;DR

This paper introduces a distillation-based approach to improve preference optimization in language models, addressing overfitting issues and enhancing robustness to distribution shifts in preference data.

Contribution

It proposes a novel distillation method that aligns implicit rewards with explicit reward models, improving robustness and avoiding overfitting in preference optimization.

Findings

Distillation from a family of reward models improves robustness to distribution shift.

The method preserves the simplicity of direct preference optimization.

Enhanced stability reduces degenerate policies and overfitting.

Abstract

Language model (LM) post-training (or alignment) involves maximizing a reward function that is derived from preference annotations. Direct Preference Optimization (DPO) is a popular offline alignment method that trains a policy directly on preference data without the need to train a reward model or apply reinforcement learning. However, the empirical evidence suggests that DPO typically assigns implicit rewards that overfit, and trend towards infinite magnitude. This frequently leads to degenerate policies, sometimes causing even the probabilities of the preferred generations to go to zero. In this work, we analyze this phenomenon and use distillation to get a better proxy for the true preference distribution over generation pairs: we train the LM such that its induced implicit reward, i.e., the scaled log-likelihood ratio of the model to the reference model, matches an explicit reward model trained on the preference data. Moreover, to account for uncertainty in the reward model we are distilling from, we optimize against a family of reward models that, as a whole, is likely to include at least one reasonable proxy for the preference distribution. Our results show that distilling from such a family of reward models leads to improved robustness to distribution shift in preference annotations, while preserving the simple supervised nature of DPO.