Rethinking DPO: The Role of Rejected Responses in Preference Misalignment

TL;DR

This paper critically examines the limitations of Direct Preference Optimization (DPO) in preference alignment and introduces Bounded-DPO (BDPO), a new method that balances the influence of rejected responses to improve alignment performance.

Contribution

The paper identifies the imbalance issue in DPO and proposes BDPO, which bounds rejected responses' influence while preserving DPO's structure, enhancing preference alignment.

Findings

BDPO outperforms existing algorithms in preference alignment tasks.

Theoretical analysis confirms BDPO's balanced optimization of responses.

Empirical results show improved generation quality with BDPO.

Abstract

Direct Preference Optimization (DPO) is a simple and efficient framework that has attracted substantial attention. However, it often struggles to meet its primary objectives -- increasing the generation probability of chosen responses while reducing that of rejected responses -- due to the dominant influence of rejected responses on the loss function. This imbalance leads to suboptimal performance in promoting preferred responses. In this work, we systematically analyze the limitations of DPO and existing algorithms designed to achieve the objectives stated above. To address these limitations, we propose Bounded-DPO (BDPO), a novel method that bounds the influence of rejected responses while maintaining the original optimization structure of DPO. Through theoretical analysis and empirical evaluations, we demonstrate that BDPO achieves a balanced optimization of the chosen and rejected responses, outperforming existing algorithms.