Why DPO is a Misspecified Estimator and How to Fix It

TL;DR

This paper analyzes the limitations of Direct Preference Optimization (DPO) as a misspecified estimator and introduces AuxDPO, an improved method that mitigates these issues for better language model alignment.

Contribution

The paper reveals DPO's misspecification problem and proposes AuxDPO, a novel method with auxiliary variables to improve alignment performance.

Findings

AuxDPO outperforms DPO in didactic bandit settings

AuxDPO achieves better alignment results on LLM tasks

DPO can cause preference reversal and reward degradation

Abstract

Direct alignment algorithms such as Direct Preference Optimization (DPO) fine-tune models based on preference data, using only supervised learning instead of two-stage reinforcement learning with human feedback (RLHF). We show that DPO encodes a statistical estimation problem over reward functions induced by a parametric policy class. When the true reward function that generates preferences cannot be realized via the policy class, DPO becomes misspecified, resulting in failure modes such as preference order reversal, worsening of policy reward, and high sensitivity to the input preference data distribution. On the other hand, we study the local behavior of two-stage RLHF for a parametric class and relate it to a natural gradient step in policy space. Our fine-grained geometric characterization allows us to propose AuxDPO, which introduces additional auxiliary variables in the DPO loss function to help move towards the RLHF solution in a principled manner and mitigate the misspecification in DPO. We empirically demonstrate the superior performance of AuxDPO on didactic bandit settings as well as LLM alignment tasks.