Multi-Step Alignment as Markov Games: An Optimistic Online Gradient Descent Approach with Convergence Guarantees

TL;DR

This paper models multi-turn language model alignment as a Markov game and introduces OMPO, an optimistic online gradient descent method with proven convergence guarantees, improving upon existing bandit-based approaches.

Contribution

It formulates the alignment problem as a multi-step Markov game and proposes OMPO, a novel algorithm with theoretical convergence guarantees for multi-turn preference optimization.

Findings

OMPO converges to an approximate Nash equilibrium within O(ε^{-1}) updates.

The method outperforms existing approaches on multi-turn conversation datasets.

Theoretical analysis confirms convergence properties of OMPO.

Abstract

Reinforcement Learning from Human Feedback (RLHF) has been highly successful in aligning large language models with human preferences. While prevalent methods like DPO have demonstrated strong performance, they frame interactions with the language model as a bandit problem, which limits their applicability in real-world scenarios where multi-turn conversations are common. Additionally, DPO relies on the Bradley-Terry model assumption, which does not adequately capture the non-transitive nature of human preferences. In this paper, we address these challenges by modeling the alignment problem as a two-player constant-sum Markov game, where each player seeks to maximize their winning rate against the other across all steps of the conversation. Our approach Optimistic Multi-step Preference Optimization (OMPO) is built upon the optimistic online mirror descent algorithm~\citep{rakhlin2013online,joulani17a}. Theoretically, we provide a rigorous analysis for the convergence of OMPO and show that OMPO requires $\mathcal{O}(\epsilon^{-1})$ policy updates to converge to an $\epsilon$-approximate Nash equilibrium. We also validate the effectiveness of our method on multi-turn conversations dataset and math reasoning dataset.