Soft Preference Optimization: Aligning Language Models to Expert Distributions

TL;DR

Soft Preference Optimization (SPO) is a novel method for aligning large language models with human preferences directly through output distributions, bypassing the need for reward models, and offering theoretical guarantees under certain assumptions.

Contribution

SPO introduces a direct output distribution optimization approach for model alignment, with a solid theoretical foundation and advantages over existing methods.

Findings

SPO converges to a softmax of scaled rewards under BT assumptions.

It offers improved alignment accuracy and computational efficiency.

The method does not require a reward model, simplifying the alignment process.

Abstract

We propose Soft Preference Optimization (SPO), a method for aligning generative models, such as Large Language Models (LLMs), with human preferences, without the need for a reward model. SPO optimizes model outputs directly over a preference dataset through a natural loss function that integrates preference loss with a regularization term across the model's entire output distribution rather than limiting it to the preference dataset. Although SPO does not require the assumption of an existing underlying reward model, we demonstrate that, under the Bradley-Terry (BT) model assumption, it converges to a softmax of scaled rewards, with the distribution's "softness" adjustable via the softmax exponent, an algorithm parameter. We showcase SPO's methodology, its theoretical foundation, and its comparative advantages in simplicity, computational efficiency, and alignment precision.