Large Language Models as Agents in Two-Player Games

TL;DR

This paper explores the analogy between large language model training processes and agent strategies in two-player games, offering new insights into LLM development, alignment, and training techniques through a game-theoretic lens.

Contribution

It introduces a novel framework re-conceptualizing LLM training as agent learning in language-based games, bridging machine learning and game theory.

Findings

Provides a new perspective on LLM training and alignment issues.

Suggests innovative data preparation and training techniques inspired by game strategies.

Unveils strategic insights for advancing LLM development.

Abstract

By formally defining the training processes of large language models (LLMs), which usually encompasses pre-training, supervised fine-tuning, and reinforcement learning with human feedback, within a single and unified machine learning paradigm, we can glean pivotal insights for advancing LLM technologies. This position paper delineates the parallels between the training methods of LLMs and the strategies employed for the development of agents in two-player games, as studied in game theory, reinforcement learning, and multi-agent systems. We propose a re-conceptualization of LLM learning processes in terms of agent learning in language-based games. This framework unveils innovative perspectives on the successes and challenges in LLM development, offering a fresh understanding of addressing alignment issues among other strategic considerations. Furthermore, our two-player game approach sheds light on novel data preparation and machine learning techniques for training LLMs.