New Desiderata for Direct Preference Optimization

TL;DR

This paper critically evaluates existing direct preference optimization (DPO) methods for aligning large language models with human preferences, identifies their limitations, and proposes an improved DPO-like loss with empirical validation.

Contribution

It introduces new evaluation criteria for DPO, highlights key shortcomings, and proposes a novel DPO-like loss that addresses these issues.

Findings

Existing DPO methods struggle with interpolation between models and preferences.

Trade-offs exist in regularization and constraint handling in current DPO approaches.

The proposed DPO-like loss mitigates identified limitations, improving alignment quality.

Abstract

Large language models in the past have typically relied on some form of reinforcement learning with human feedback (RLHF) to better align model responses with human preferences. However, because of oft-observed instabilities when implementing these RLHF pipelines, various reparameterization techniques have recently been introduced to sidestep the need for separately learning an RL reward model. Instead, directly fine-tuning for human preferences is achieved via the minimization of a single closed-form training objective, a process originally referred to as direct preference optimization (DPO) and followed by several notable descendants. Although effective in certain real-world settings, we introduce new evaluation criteria that serve to highlight unresolved shortcomings in the ability of existing DPO methods to interpolate between a pre-trained reference model and empirical measures of human preferences, as well as unavoidable trade-offs in how low- and high-quality responses are regularized and constraints are handled. Our insights then motivate an alternative DPO-like loss that provably mitigates these limitations. Empirical results serve to corroborate notable aspects of our analyses.