The Crucial Role of Samplers in Online Direct Preference Optimization

TL;DR

This paper analyzes how different sampling strategies affect the convergence of Direct Preference Optimization (DPO) in language model alignment, revealing that tailored samplers significantly improve convergence rates and practical performance.

Contribution

The paper provides a rigorous theoretical analysis of DPO's convergence with various samplers and introduces an improved online sampler that enhances real-world performance.

Findings

Uniform sampling achieves linear convergence.

Proposed online sampler achieves quadratic convergence.

Outperforms vanilla DPO by over 7.4% on Safe-RLHF dataset.

Abstract

Direct Preference Optimization (DPO) has emerged as a stable, scalable, and efficient solution for language model alignment. Despite its empirical success, the optimization properties, particularly the impact of samplers on its convergence rates, remain under-explored. In this paper, we provide a rigorous analysis of DPO's convergence rates with different sampling strategies under the exact gradient setting, revealing a surprising separation: uniform sampling achieves $\textbf{linear}$ convergence, while our proposed online sampler achieves $\textbf{quadratic}$ convergence. We further adapt the sampler to practical settings by incorporating posterior distributions and logit mixing, demonstrating improvements over previous methods. For example, it outperforms vanilla DPO by over $7.4$% on Safe-RLHF dataset. Our results not only offer insights into the theoretical understanding of DPO but also pave the way for further algorithm designs.