Distributional Preference Alignment of LLMs via Optimal Transport

TL;DR

This paper introduces Alignment via Optimal Transport (AOT), a novel distributional preference alignment method for LLMs that improves alignment quality by optimizing the reward distribution dominance using a convex optimal transport formulation.

Contribution

The paper proposes a new distributional alignment method for LLMs using optimal transport, with a convex relaxation and closed-form solution, achieving state-of-the-art results.

Findings

AOT achieves state-of-the-art performance on multiple alignment datasets.

The method converges at the parametric rate based on sample complexity analysis.

Empirical results show improved alignment in 7B LLMs on benchmark evaluations.

Abstract

Current LLM alignment techniques use pairwise human preferences at a sample level, and as such, they do not imply an alignment on the distributional level. We propose in this paper Alignment via Optimal Transport (AOT), a novel method for distributional preference alignment of LLMs. AOT aligns LLMs on unpaired preference data by making the reward distribution of the positive samples stochastically dominant in the first order on the distribution of negative samples. We introduce a convex relaxation of this first-order stochastic dominance and cast it as an optimal transport problem with a smooth and convex cost. Thanks to the one-dimensional nature of the resulting optimal transport problem and the convexity of the cost, it has a closed-form solution via sorting on empirical measures. We fine-tune LLMs with this AOT objective, which enables alignment by penalizing the violation of the stochastic dominance of the reward distribution of the positive samples on the reward distribution of the negative samples. We analyze the sample complexity of AOT by considering the dual of the OT problem and show that it converges at the parametric rate. Empirically, we show on a diverse set of alignment datasets and LLMs that AOT leads to state-of-the-art models in the 7B family of models when evaluated with Open LLM Benchmarks and AlpacaEval.