Enhance Reasoning for Large Language Models in the Game Werewolf

TL;DR

This paper introduces a hierarchical reasoning framework combining LLMs with an external Thinker module to improve complex reasoning in social deduction games like Werewolf, outperforming existing models including GPT-4.

Contribution

It presents a novel hierarchical reasoning framework integrating LLMs with an external Thinker, trained on extensive data, to enhance reasoning and outperform GPT-4 in social deduction tasks.

Findings

Framework improves deductive reasoning and speech generation.

Fine-tuned 6B LLM surpasses GPT-4 with Thinker integration.

Introduces largest social deduction game dataset to date.

Abstract

This paper presents an innovative framework that integrates Large Language Models (LLMs) with an external Thinker module to enhance the reasoning capabilities of LLM-based agents. Unlike augmenting LLMs with prompt engineering, Thinker directly harnesses knowledge from databases and employs various optimization techniques. The framework forms a reasoning hierarchy where LLMs handle intuitive System-1 tasks such as natural language processing, while the Thinker focuses on cognitive System-2 tasks that require complex logical analysis and domain-specific knowledge. Our framework is presented using a 9-player Werewolf game that demands dual-system reasoning. We introduce a communication protocol between LLMs and the Thinker, and train the Thinker using data from 18800 human sessions and reinforcement learning. Experiments demonstrate the framework's effectiveness in deductive reasoning, speech generation, and online game evaluation. Additionally, we fine-tune a 6B LLM to surpass GPT4 when integrated with the Thinker. This paper also contributes the largest dataset for social deduction games to date.