Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO

TL;DR

This paper introduces PRO, a new alignment method for large language models that effectively handles diverse feedback types and addresses the limitations of traditional contrastive alignment methods.

Contribution

The paper provides a principled decomposition of DPO, identifies the cause of likelihood underdetermination, and proposes PRO, a unified method that improves alignment with various feedback types.

Findings

PRO outperforms existing methods on multiple feedback types

Restoring the full regularizer resolves likelihood underdetermination

PRO demonstrates consistent improvements in empirical evaluations

Abstract

Direct alignment methods typically train large language models (LLMs) by contrasting the likelihoods of preferred and dispreferred responses. While effective at capturing relative preferences, these methods are widely observed to suppress the absolute likelihoods of example responses. As a result, aligned models can deviate from expected patterns, exhibiting rewar-hacking effect even without an explicit reward model. This fundamental limitation of contrastive alignment, which we term likelihood underdetermination, motivates us to revisit direct preference optimization (DPO) -- the seminal direct alignment method. Interestingly, we show that the DPO loss admits a principled decomposition. The reformulated loss not only extends naturally to a broader range of feedback types, but also unveils the root cause of likelihood underdetermination. Specifically, we identify that standard DPO implicitly oversimplifies a regularizer in the reformulated loss; restoring this full term effectively resolves the underdetermination. Building on these insights, we introduce PRoximalized PReference Optimization (PRO), a unified alignment method that accommodates diverse feedback types while eliminating likelihood underdetermination through an efficient approximation of the full regularizer. Empirical evaluations demonstrate the consistent superiority of PRO over existing methods across pairwise, binary and scalar feedback.