Magnetic Preference Optimization: Achieving Last-iterate Convergence for Language Model Alignment

TL;DR

This paper introduces Magnetic Preference Optimization (MPO), a new method that guarantees last-iterate convergence to the Nash equilibrium in preference-based games, improving language model alignment with theoretical guarantees and practical efficiency.

Contribution

MPO extends Magnetic Mirror Descent to preference optimization, achieving last-iterate convergence and linear rates, addressing limitations of existing methods in LLM fine-tuning.

Findings

MPO achieves last-iterate convergence to the NE.

MPO demonstrates linear convergence rate.

Empirical results show improved LLM performance.

Abstract

Self-play methods have demonstrated remarkable success in enhancing model capabilities across various domains. In the context of Reinforcement Learning from Human Feedback (RLHF), self-play not only boosts Large Language Model (LLM) performance but also overcomes the limitations of traditional Bradley-Terry (BT) model assumptions by finding the Nash equilibrium (NE) of a preference-based, two-player constant-sum game. However, existing methods either guarantee only average-iterate convergence, incurring high storage and inference costs, or converge to the NE of a regularized game, failing to accurately reflect true human preferences. In this paper, we introduce Magnetic Preference Optimization (MPO), a novel approach capable of achieving last-iterate convergence to the NE of the original game, effectively overcoming the limitations of existing methods. Building upon Magnetic Mirror Descent (MMD), MPO attains a linear convergence rate, making it particularly suitable for fine-tuning LLMs. To ensure our algorithm is both theoretically sound and practically viable, we present a simple yet effective implementation that adapts the theoretical insights to the RLHF setting. Empirical results demonstrate that MPO can significantly enhance the performance of LLMs, highlighting the potential of self-play methods in alignment.