$\xi$-DPO: Direct Preference Optimization via Ratio Reward Margin

TL;DR

The paper introduces $\xi$-DPO, a novel preference optimization method that reformulates the objective to improve interpretability and eliminate hyperparameter tuning challenges in reference-free preference learning.

Contribution

It proposes a new ratio reward margin formulation that simplifies preference optimization and removes the need for tuning hyperparameters like $eta$ and $\gamma$.

Findings

$\xi$-DPO effectively cancels the effect of $eta$ in preference optimization.

The ratio reward margin $\xi$ is interpretable and can be set based on initial reward gap distribution.

Experimental results demonstrate improved stability and performance over existing methods.

Abstract

Reference-free preference optimization has emerged as an efficient alternative to reinforcement learning from human feedback, with Simple Preference Optimization(SimPO) demonstrating strong performance by eliminating the explicit reference model through a simple objective. However, the joint tuning of the hyperparameters $\beta$ and $\gamma$ in SimPO remains a central challenge. We argue that this difficulty arises because the margin formulation in SimPO is not easily interpretable across datasets with different reward gap structures. To better understand this issue, we conduct a comprehensive analysis of SimPO and find that $\beta$ implicitly controls sample filtering, while the effect of $\gamma$ depends on the reward gap structure of the dataset. Motivated by these observations, we propose $\xi$-DPO: Direct preference optimization via ratio reward margin. We first reformulate the preference objective through an equivalent transformation, changing the optimization target from maximizing the likelihood of reward gaps to minimizing the distance between reward gaps and optimal margins. Then, we redefine the reward in a ratio form between the chosen and rejected, which effectively cancels the effect of $\beta$ and yields a bounded and interpretable margin. This margin is called the ratio reward margin and is denoted by $\xi$. Unlike the margin $\gamma$ in SimPO, $\xi$ explicitly represents the desired relative separation between chosen and rejected responses and can be determined from the initial reward gap distribution, avoiding repeated trial-and-error tuning. ....