Strategist: Self-improvement of LLM Decision Making via Bi-Level Tree Search

TL;DR

STRATEGIST is a novel framework combining LLMs and Monte Carlo Tree Search to improve decision-making in complex, multi-turn games without training data, outperforming traditional RL methods and matching human performance.

Contribution

It introduces a generalizable, training-free approach that integrates LLMs with tree search to optimize strategies through self-play in complex games.

Findings

Outperforms traditional RL agents in multi-turn, partial information games.

Achieves competitive performance against human players.

Effective in learning strategies without any training data.

Abstract

Traditional reinforcement learning and planning typically requires vast amounts of data and training to develop effective policies. In contrast, large language models (LLMs) exhibit strong generalization and zero-shot capabilities, but struggle with tasks that require detailed planning and decision-making in complex action spaces. We introduce STRATEGIST, a novel approach that integrates the strengths of both methods. Our approach leverages LLMs to search and update high-level strategies (as text), which are then refined and executed by low-level Monte Carlo Tree Search (MCTS). STRATEGIST is a generalizable framework to optimize the strategy through population-based self-play simulations without the need for any training data. We demonstrate the effectiveness of STRATEGIST in learning optimal strategies for competitive, multi-turn games with partial information, including Game of Pure Strategy (GOPS) and multi-agent, hidden-identity discussion games like The Resistance: Avalon. Our results show that agents equipped with STRATEGIST outperform those trained with traditional RL methods, other LLM-based skill acquisition techniques, pre-existing LLM agents across both game environments and achieves comparable performance against human players.