Transformer Based Planning in the Observation Space with Applications to Trick Taking Card Games

TL;DR

This paper introduces GO-MCTS, a novel planning algorithm using transformers for generative modeling in the observation space, improving decision-making in imperfect information trick-taking card games.

Contribution

It presents a new MCTS variant that leverages transformers as generative models for observation sequences, tailored for imperfect information games.

Findings

GO-MCTS outperforms traditional methods in multiple card games.

Transformers effectively model observation sequences for planning.

Iterative training via self-play enhances the model's performance.

Abstract

Traditional search algorithms have issues when applied to games of imperfect information where the number of possible underlying states and trajectories are very large. This challenge is particularly evident in trick-taking card games. While state sampling techniques such as Perfect Information Monte Carlo (PIMC) search has shown success in these contexts, they still have major limitations. We present Generative Observation Monte Carlo Tree Search (GO-MCTS), which utilizes MCTS on observation sequences generated by a game specific model. This method performs the search within the observation space and advances the search using a model that depends solely on the agent's observations. Additionally, we demonstrate that transformers are well-suited as the generative model in this context, and we demonstrate a process for iteratively training the transformer via population-based self-play. The efficacy of GO-MCTS is demonstrated in various games of imperfect information, such as Hearts, Skat, and "The Crew: The Quest for Planet Nine," with promising results.