Safe and Nested Subgame Solving for Imperfect-Information Games

TL;DR

This paper introduces advanced subgame-solving techniques for imperfect-information games, enabling more accurate and adaptive strategies that outperform previous methods and contributed to AI success in poker.

Contribution

It presents novel subgame-solving methods that improve theoretical and practical performance, including adaptation to outside actions and iterative solving during gameplay.

Findings

Outperforms prior subgame-solving methods in theory and practice

Adapts to opponent actions outside original abstractions

Enables iterative solving to reduce exploitability

Abstract

In imperfect-information games, the optimal strategy in a subgame may depend on the strategy in other, unreached subgames. Thus a subgame cannot be solved in isolation and must instead consider the strategy for the entire game as a whole, unlike perfect-information games. Nevertheless, it is possible to first approximate a solution for the whole game and then improve it by solving individual subgames. This is referred to as subgame solving. We introduce subgame-solving techniques that outperform prior methods both in theory and practice. We also show how to adapt them, and past subgame-solving techniques, to respond to opponent actions that are outside the original action abstraction; this significantly outperforms the prior state-of-the-art approach, action translation. Finally, we show that subgame solving can be repeated as the game progresses down the game tree, leading to far lower exploitability. These techniques were a key component of Libratus, the first AI to defeat top humans in heads-up no-limit Texas hold'em poker.