Direct Preference Optimization with Rating Information: Practical Algorithms and Provable Gains

TL;DR

This paper introduces algorithms that leverage rating gap information to improve preference optimization in language models, achieving faster learning and robustness to inaccuracies, with strong empirical results across various benchmarks.

Contribution

The paper proposes new algorithms that utilize rating gap data for preference optimization, providing theoretical and empirical advantages over existing DPO methods.

Findings

Faster statistical convergence with accurate rating gaps.

Robustness of algorithms to rating gap inaccuracies.

Superior performance across multiple LLM benchmarks.

Abstract

The class of direct preference optimization (DPO) algorithms has emerged as a promising approach for solving the alignment problem in foundation models. These algorithms work with very limited feedback in the form of pairwise preferences and fine-tune models to align with these preferences without explicitly learning a reward model. While the form of feedback used by these algorithms makes the data collection process easy and relatively more accurate, its ambiguity in terms of the quality of responses could have negative implications. For example, it is not clear if a decrease (increase) in the likelihood of preferred (dispreferred) responses during the execution of these algorithms could be interpreted as a positive or negative phenomenon. In this paper, we study how to design algorithms that can leverage additional information in the form of rating gap, which informs the learner how much the chosen response is better than the rejected one. We present new algorithms that can achieve faster statistical rates than DPO in presence of accurate rating gap information. Moreover, we theoretically prove and empirically show that the performance of our algorithms is robust to inaccuracy in rating gaps. Finally, we demonstrate the solid performance of our methods in comparison to a number of DPO-style algorithms across a wide range of LLMs and evaluation benchmarks.