Self-playing Adversarial Language Game Enhances LLM Reasoning

TL;DR

This paper introduces a self-play adversarial language game called SPAG to enhance large language models' reasoning abilities, demonstrating consistent performance improvements across various benchmarks through reinforcement learning.

Contribution

It presents a novel self-play training framework for LLMs using an adversarial language game, significantly improving their reasoning skills.

Findings

LLMs' reasoning performance improves after self-play training

Self-play leads to continuous enhancement of reasoning abilities

The method achieves broad improvements across multiple benchmarks

Abstract

We explore the potential of self-play training for large language models (LLMs) in a two-player adversarial language game called Adversarial Taboo. In this game, an attacker and a defender communicate around a target word only visible to the attacker. The attacker aims to induce the defender to speak the target word unconsciously, while the defender tries to infer the target word from the attacker's utterances. To win the game, both players must have sufficient knowledge about the target word and high-level reasoning ability to infer and express in this information-reserved conversation. Hence, we are curious about whether LLMs' reasoning ability can be further enhanced by Self-Playing this Adversarial language Game (SPAG). With this goal, we select several open-source LLMs and let each act as the attacker and play with a copy of itself as the defender on an extensive range of target words. Through reinforcement learning on the game outcomes, we observe that the LLMs' performances uniformly improve on a broad range of reasoning benchmarks. Furthermore, iteratively adopting this self-play process can continuously promote LLMs' reasoning abilities. The code is available at https://github.com/Linear95/SPAG.