ExIt-OOS: Towards Learning from Planning in Imperfect Information Games

TL;DR

This paper introduces ExIt-OOS, a new method combining online outcome sampling with deep reinforcement learning within the Expert Iteration framework to improve performance in imperfect information games.

Contribution

It extends AlphaZero-like approaches to imperfect information games by integrating Online Outcome Sampling into the training and planning process.

Findings

Effective integration of OOS with neural strategies

Improved performance in imperfect information games

Establishment of a learning and planning feedback loop

Abstract

The current state of the art in playing many important perfect information games, including Chess and Go, combines planning and deep reinforcement learning with self-play. We extend this approach to imperfect information games and present ExIt-OOS, a novel approach to playing imperfect information games within the Expert Iteration framework and inspired by AlphaZero. We use Online Outcome Sampling, an online search algorithm for imperfect information games in place of MCTS. While training online, our neural strategy is used to improve the accuracy of playouts in OOS, allowing a learning and planning feedback loop for imperfect information games.