Coverage Improvement and Fast Convergence of On-policy Preference Learning

TL;DR

This paper provides a theoretical analysis of on-policy preference learning algorithms, demonstrating their rapid convergence due to coverage improvement, and introduces new methods that outperform off-policy approaches in language model alignment.

Contribution

It introduces the coverage improvement principle, proves exponential convergence of on-policy DPO under certain conditions, and proposes a hybrid sampling method for faster convergence.

Findings

On-policy DPO converges exponentially with sufficient batch size.

Hybrid sampler guarantees convergence in just two rounds.

On-policy methods outperform off-policy counterparts in experiments.

Abstract

Online on-policy preference learning algorithms for language model alignment such as online direct policy optimization (DPO) can significantly outperform their offline counterparts. We provide a theoretical explanation for this phenomenon by analyzing how the sampling policy's coverage evolves throughout on-policy training. We propose and rigorously justify the \emph{coverage improvement principle}: with sufficient batch size, each update moves into a region around the target where coverage is uniformly better, making subsequent data increasingly informative and enabling rapid convergence. In the contextual bandit setting with Bradley-Terry preferences and linear softmax policy class, we show that on-policy DPO converges exponentially in the number of iterations for batch size exceeding a generalized coverage threshold. In contrast, any learner restricted to offline samples from the initial policy suffers a slower minimax rate, leading to a sharp separation in total sample complexity. Motivated by this analysis, we further propose a simple hybrid sampler based on a novel \emph{preferential} G-optimal design, which removes dependence on coverage and guarantees convergence in just two rounds. Finally, we develop principled on-policy schemes for reward distillation in the general function class setting, and show faster noiseless rates under an alternative deviation-based notion of coverage. Experimentally, we confirm that on-policy DPO and our proposed reward distillation algorithms outperform their off-policy counterparts and enjoy stable, monotonic performance gains across iterations.