Mastering Board Games by External and Internal Planning with Language Models

TL;DR

This paper demonstrates that search-based planning methods, both external and internal, significantly enhance the game-playing strength of Large Language Models across various board games, achieving near-human and Grandmaster-level performance.

Contribution

The paper introduces and compares two novel search-based planning approaches for LLMs in board games, achieving state-of-the-art performance without external game engines.

Findings

LLMs with search outperform base models in multiple board games.

Achieved Grandmaster-level performance in chess with LLM-based planning.

Both approaches operate efficiently with minimal hallucinations.

Abstract

Advancing planning and reasoning capabilities of Large Language Models (LLMs) is one of the key prerequisites towards unlocking their potential for performing reliably in complex and impactful domains. In this paper, we aim to demonstrate this across board games (Chess, Fischer Random / Chess960, Connect Four, and Hex), and we show that search-based planning can yield significant improvements in LLM game-playing strength. We introduce, compare and contrast two major approaches: In external search, the model guides Monte Carlo Tree Search (MCTS) rollouts and evaluations without calls to an external game engine, and in internal search, the model is trained to generate in-context a linearized tree of search and a resulting final choice. Both build on a language model pre-trained on relevant domain knowledge, reliably capturing the transition and value functions in the respective environments, with minimal hallucinations. We evaluate our LLM search implementations against game-specific state-of-the-art engines, showcasing substantial improvements in strength over the base model, and reaching Grandmaster-level performance in chess while operating closer to the human search budget. Our proposed approach, combining search with domain knowledge, is not specific to board games, hinting at more general future applications.