Learning to Discuss Strategically: A Case Study on One Night Ultimate Werewolf

TL;DR

This paper investigates strategic discussion in the game One Night Ultimate Werewolf, analyzing equilibrium strategies and proposing a reinforcement learning-based language agent to optimize discussion tactics, demonstrating improved performance.

Contribution

It provides the first analysis of perfect Bayesian equilibria in ONUW with and without discussion and introduces a RL-instructed language agent for strategic communication.

Findings

Discussion significantly impacts players' utilities and beliefs.

The RL-based agent effectively learns and generalizes discussion tactics.

Experimental results show the framework's effectiveness across game settings.

Abstract

Communication is a fundamental aspect of human society, facilitating the exchange of information and beliefs among people. Despite the advancements in large language models (LLMs), recent agents built with these often neglect the control over discussion tactics, which are essential in communication scenarios and games. As a variant of the famous communication game Werewolf, One Night Ultimate Werewolf (ONUW) requires players to develop strategic discussion policies due to the potential role changes that increase the uncertainty and complexity of the game. In this work, we first present the existence of the Perfect Bayesian Equilibria (PBEs) in two scenarios of the ONUW game: one with discussion and one without. The results showcase that the discussion greatly changes players' utilities by affecting their beliefs, emphasizing the significance of discussion tactics. Based on the insights obtained from the analyses, we propose an RL-instructed language agent framework, where a discussion policy trained by reinforcement learning (RL) is employed to determine appropriate discussion tactics to adopt. Our experimental results on several ONUW game settings demonstrate the effectiveness and generalizability of our proposed framework. The project page of our paper: $\href{https://one-night-ultimate-werewolf.github.io}{one-night-ultimate-werewolf.github.io}$.