Your Self-Play Algorithm is Secretly an Adversarial Imitator: Understanding LLM Self-Play through the Lens of Imitation Learning

TL;DR

This paper reveals that self-play finetuning of large language models can be understood as adversarial imitation learning, providing a theoretical framework and a new stable algorithm that improves performance across tasks.

Contribution

It introduces a novel game-theoretic perspective linking self-play to adversarial imitation learning and proposes a new finetuning algorithm based on $\\chi^2$-divergence for better stability.

Findings

Convergence of self-play finetuning to equilibrium shown theoretically.

The proposed algorithm outperforms existing self-play methods.

Experimental validation across multiple language tasks.

Abstract

Self-play post-training methods has emerged as an effective approach for finetuning large language models and turn the weak language model into strong language model without preference data. However, the theoretical foundations for self-play finetuning remain underexplored. In this work, we tackle this by connecting self-play finetuning with adversarial imitation learning by formulating finetuning procedure as a min-max game between the model and a regularized implicit reward player parameterized by the model itself. This perspective unifies self-play imitation and general preference alignment within a common framework. Under this formulation, we present a game-theoretic analysis showing that the self-play finetuning will converge to it's equilibrium. Guided by this theoretical formulation, we propose a new self-play imitation finetuning algorithm based on the $\chi^2$-divergence variational objective with bounded rewards and improved stability. Experiments on various of language model finetuning tasks demonstrate consistent improvements over existing self-play methods and validate our theoretical insights.