Learning Strategic Language Agents in the Werewolf Game with Iterative Latent Space Policy Optimization

TL;DR

This paper introduces LSPO, an iterative framework combining game theory and LLM fine-tuning to develop strategic language agents for the Werewolf game, significantly improving their reasoning and communication capabilities.

Contribution

The paper presents a novel Latent Space Policy Optimization method that effectively manages large language spaces and enhances strategic decision-making in language-based games.

Findings

LSPO agents outperform existing Werewolf agents.

Iterative strategy space expansion improves performance.

Combines game-theoretic optimization with LLM fine-tuning.

Abstract

Large language model (LLM) agents have recently demonstrated impressive capabilities in various domains like open-ended conversation and multi-step decision-making. However, it remains challenging for these agents to solve strategic language games, such as Werewolf, which demand both strategic decision-making and free-form language interactions. Existing LLM agents often suffer from intrinsic bias in their action distributions and limited exploration of the unbounded text action space, resulting in suboptimal performance. To address these challenges, we propose Latent Space Policy Optimization (LSPO), an iterative framework that combines game-theoretic methods with LLM fine-tuning to build strategic language agents. LSPO leverages the observation that while the language space is combinatorially large, the underlying strategy space is relatively compact. We first map free-form utterances into a finite latent strategy space, yielding an abstracted extensive-form game. Then we apply game-theoretic methods like Counterfactual Regret Minimization (CFR) to optimize the policy in the latent space. Finally, we fine-tune the LLM via Direct Preference Optimization (DPO) to align with the learned policy. By iteratively alternating between these steps, our LSPO agents progressively enhance both strategic reasoning and language communication. Experiment on the Werewolf game shows that our agents iteratively expand the strategy space with improving performance and outperform existing Werewolf agents, underscoring their effectiveness in free-form language games with strategic interactions.