Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback

TL;DR

This paper systematically investigates how different components of preference-based learning affect language model performance, highlighting the importance of data quality, algorithms, reward models, and prompts, with practical recommendations and empirical results.

Contribution

It provides a comprehensive analysis of preference learning components, compares DPO and PPO, and offers a recipe for effective preference-based training of language models.

Findings

Preference data quality has the largest impact on performance.

PPO outperforms DPO by up to 2.5% in math domains.

High-quality preference data improves instruction following by up to 8%.

Abstract

Learning from preference feedback has emerged as an essential step for improving the generation quality and performance of modern language models (LMs). Despite its widespread use, the way preference-based learning is applied varies wildly, with differing data, learning algorithms, and evaluations used, making disentangling the impact of each aspect difficult. In this work, we identify four core aspects of preference-based learning: preference data, learning algorithm, reward model, and policy training prompts, systematically investigate the impact of these components on downstream model performance, and suggest a recipe for strong learning for preference feedback. Our findings indicate that all aspects are important for performance, with better preference data leading to the largest improvements, followed by the choice of learning algorithm, the use of improved reward models, and finally the use of additional unlabeled prompts for policy training. Notably, PPO outperforms DPO by up to 2.5% in math and 1.2% in general domains. High-quality preference data leads to improvements of up to 8% in instruction following and truthfulness. Despite significant gains of up to 5% in mathematical evaluation when scaling up reward models, we surprisingly observe marginal improvements in other categories. We publicly release the code used for training (https://github.com/hamishivi/EasyLM) and evaluating (https://github.com/allenai/open-instruct) our models, along with the models and datasets themselves (https://huggingface.co/collections/allenai/tulu-v25-suite-66676520fd578080e126f618).