Energy-Based Preference Model Offers Better Offline Alignment than the Bradley-Terry Preference Model

TL;DR

This paper introduces an energy-based preference model with a contrastive loss function that ensures a unique maximum likelihood estimator, leading to improved offline alignment of language models compared to the Bradley-Terry preference model.

Contribution

The paper proposes an energy-based preference model and a contrastive loss (EPA) that guarantees a unique MLE, addressing limitations of the Bradley-Terry model in offline alignment tasks.

Findings

EPA outperforms DPO on open benchmarks

Energy-based model ensures a unique MLE

Contrastive loss reduces approximation error

Abstract

Since the debut of DPO, it has been shown that aligning a target LLM with human preferences via the KL-constrained RLHF loss is mathematically equivalent to a special kind of reward modeling task. Concretely, the task requires: 1) using the target LLM to parameterize the reward model, and 2) tuning the reward model so that it has a 1:1 linear relationship with the true reward. However, we identify a significant issue: the DPO loss might have multiple minimizers, of which only one satisfies the required linearity condition. The problem arises from a well-known issue of the underlying Bradley-Terry preference model: it does not always have a unique maximum likelihood estimator (MLE). Consequently,the minimizer of the RLHF loss might be unattainable because it is merely one among many minimizers of the DPO loss. As a better alternative, we propose an energy-based model (EBM) that always has a unique MLE, inherently satisfying the linearity requirement. To approximate the MLE in practice, we propose a contrastive loss named Energy Preference Alignment (EPA), wherein each positive sample is contrasted against one or more strong negatives as well as many free weak negatives. Theoretical properties of our EBM enable the approximation error of EPA to almost surely vanish when a sufficient number of negatives are used. Empirically, we demonstrate that EPA consistently delivers better performance on open benchmarks compared to DPO, thereby showing the superiority of our EBM.