What Is Preference Optimization Doing, and Why?

TL;DR

This paper analyzes the optimization dynamics of preference optimization methods like DPO and PPO in large language models, revealing their distinct behaviors and roles of key components to improve understanding and development.

Contribution

It provides a detailed analysis of the underlying causes of differences between DPO and PPO, offering new insights into their optimization dynamics and roles of components.

Findings

DPO follows stable target directions, while PPO balances exploration and exploitation.

Loss reweighting in DPO acts as a regularizer, not a reward signal.

Negative learning in PPO primarily supports exploration.

Abstract

Preference optimization (PO) is indispensable for large language models (LLMs), with methods such as direct preference optimization (DPO) and proximal policy optimization (PPO) achieving great success. A common belief is that DPO is supervised learning while PPO is reinforcement learning, yet deeper analyses for the reasons underlying these differences remain lacking. To fill this gap, we analyze their optimization dynamics, revealing distinct algorithmic behaviors and comprehending their underlying causes. First, we examine the target directions of gradient-based updates and find that DPO follows stable targets, whereas PPO balances exploration and exploitation, validating the common belief yet from this new perspective. Second, we examine the roles of positive learning, negative learning, and loss reweighting, which are three key yet seldom discussed components within PO methods. Our analyses reveal that these components play fairly different roles. In DPO, positive and negative learning jointly shape the targets. However, loss reweighting in DPO acts less as a reward signal but more as a regularizer to mitigate overfitting. In PPO, negative learning primarily supports exploration rather than determining the targets. Meanwhile, loss reweighting, related to the absolute advantages, indicates the distinct roles of token groups in updating targets. Given these findings, we conduct carefully designed ablation studies to further examine how controlling these dynamics impacts optimization efficiency and practical performance. The insights gained from our analyses not only deepen the understanding of PO methods but also inspire the development of more preference-aligned LLMs.