Listwise Direct Preference Optimization with Multi-Dimensional Preference Mixing

TL;DR

This paper introduces $$-DPO, a unified framework for preference optimization that models multi-dimensional human preferences using a mixture of listwise distributions, improving flexibility and robustness in preference learning.

Contribution

The paper proposes $$-DPO, a novel method that captures multi-dimensional preferences with a mixture model and an adaptive scheduler, enhancing preference modeling and robustness.

Findings

Consistent performance improvements across multiple benchmarks.

Effective modeling of multi-dimensional human preferences.

Robustness gained through adaptive preference weighting.

Abstract

Recent alignment methods based on Direct Preference Optimization (DPO) reformulate preference learning as supervised optimization over pairwise comparisons, offering improved efficiency and stability over reinforcement learning from human feedback (RLHF). However, existing DPO-style methods implicitly assume a single fixed preference objective, which limits their ability to model the structured and sometimes conflicting nature of real-world human judgments that span multiple preference dimensions. In this work, we propose Listwise Direct Preference Optimization ($\lambda$-DPO), a unified framework that simultaneously improves supervision granularity and preference flexibility. Instead of collapsing multi-dimensional preference signals into a single ranking, $\lambda$-DPO constructs a mixture of listwise preference distributions weighted by a preference vector $\lambda$ on the probability simplex, enabling a single model to internalize a continuous spectrum of preference trade-offs. To further improve robustness, we introduce a performance-driven stochastic $\lambda$ scheduler that adaptively samples preference weights based on empirical downstream performance, explicitly mitigating the risks of misspecification inherent to static weighting schemes. We evaluate our method across multiple model families and scales on six widely used benchmarks. Experimental results show the consistent improvement against baselines.