C2-DPO: Constrained Controlled Direct Preference Optimization

TL;DR

This paper introduces C2-DPO, a constrained version of DPO, which improves language model alignment by controlling probability mass displacement, supported by theoretical insights and practical experiments.

Contribution

We derive a new perspective on DPO, propose constraints to control probability shifts, and demonstrate improved alignment results with C2-DPO.

Findings

C2-DPO outperforms vanilla DPO in language model alignment tasks.

Theoretical analysis links C2-DPO to RLHF principles.

Constraints effectively limit probability mass displacement.

Abstract

Direct preference optimization (\texttt{DPO}) has emerged as a promising approach for solving the alignment problem in AI. In this paper, we make two counter-intuitive observations about \texttt{DPO}. First, we show that \texttt{DPO} loss could be derived by starting from an alternative optimization problem that only defines the KL guardrail on in-sample responses, unlike the original RLHF problem where guardrails are defined on the entire distribution. Second, we prove a surprising property of this alternative optimization problem, namely that under its optimal policy, both preferred and rejected responses tend to decrease in probability, a phenomenon typically displayed by DPO in practice. To control this behavior, we propose a set of constraints designed to limit the displacement of probability mass between the preferred and rejected responses in the reference and target policies. The resulting algorithm, which we call Constrained Controlled DPO (\texttt{C2-DPO}), has a meaningful RLHF interpretation. By hedging against the displacement, \texttt{C2-DPO} provides practical improvements over vanilla \texttt{DPO} when aligning several language models using standard preference datasets.