Explicit Preference Optimization: No Need for an Implicit Reward Model

TL;DR

This paper introduces EXPO, an explicit preference optimization framework for training large language models that avoids the pitfalls of implicit reward reparameterizations used in prior methods like DPO, leading to more transparent and effective preference alignment.

Contribution

The paper proposes EXPO, a novel explicit preference optimization method that eliminates the need for reparameterized implicit rewards, addressing limitations of existing DPO-based approaches.

Findings

EXPO outperforms DPO in preference alignment tasks.

EXPO demonstrates more transparent regularization and avoids counter-intuitive behaviors.

Empirical results validate the theoretical advantages of EXPO over implicit reward methods.

Abstract

The generated responses of large language models (LLMs) are often fine-tuned to human preferences through a process called reinforcement learning from human feedback (RLHF). As RLHF relies on a challenging training sequence, whereby a separate reward model is independently learned and then later applied to LLM policy updates, ongoing research effort has targeted more straightforward alternatives. In this regard, direct preference optimization (DPO) and its many offshoots circumvent the need for a separate reward training step. Instead, through the judicious use of a reparameterization trick that induces an \textit{implicit} reward, DPO and related methods consolidate learning to the minimization of a single loss function. And yet despite demonstrable success in some real-world settings, we prove that DPO-based objectives are nonetheless subject to sub-optimal regularization and counter-intuitive interpolation behaviors, underappreciated artifacts of the reparameterizations upon which they are based. To this end, we introduce an \textit{explicit} preference optimization framework termed EXPO that requires no analogous reparameterization to achieve an implicit reward. Quite differently, we merely posit intuitively-appealing regularization factors from scratch that transparently avoid the potential pitfalls of key DPO variants, provably satisfying regularization desiderata that prior methods do not. Empirical results serve to corroborate our analyses and showcase the efficacy of EXPO.