Efficacy of Language Model Self-Play in Non-Zero-Sum Games

TL;DR

This paper explores the use of self-play to improve language models in negotiation games, demonstrating significant performance gains and generalization to human collaboration across cooperative and competitive settings.

Contribution

It empirically shows that language models can effectively use self-play to improve in negotiation tasks and generalize to human interactions, even in cooperative scenarios.

Findings

Models improve 14-17x in task reward after self-play finetuning.

Trained models outperform base models 2.5-6x in human collaboration.

Self-play enhances language model performance in both cooperative and competitive settings.

Abstract

Game-playing agents like AlphaGo have achieved superhuman performance through self-play, which is theoretically guaranteed to yield optimal policies in competitive games. However, most language tasks are partially or fully cooperative, so it is an open question whether techniques like self-play can effectively be used to improve language models. We empirically investigate this question in a negotiation game setting known as Deal or No Deal (DoND). Crucially, the objective in DoND can be modified to produce a fully cooperative game, a strictly competitive one, or anything in between. We finetune language models in self-play over multiple rounds of filtered behavior cloning in DoND for each of these objectives and evaluate them in self-play and in collaboration with humans. We find that language models improve substantially in self-play, achieving 14-17x higher scores in task reward after finetuning. Further, the trained models generalize to both cooperation and competition with humans, scoring 2.5-6x higher than base models. We view these results as an early promising sign for language model self-play in cooperative settings, despite a lack of theoretical guarantees.