Equilibrium Refinements Improve Subgame Solving in Imperfect-Information Games

TL;DR

This paper introduces equilibrium refinements for subgame solving in imperfect-information games, demonstrating that refined equilibria significantly outperform unrefined ones and reduce exploitability.

Contribution

It proposes gadget game sequential equilibria and modifications to solution algorithms, improving robustness and performance in imperfect-information game subgame solving.

Findings

Refined equilibria outperform unrefined Nash equilibria in benchmarks.

Strategies with refinements reduce exploitability by over 50%.

Gadget game sequential equilibria are computationally feasible with mild overhead.

Abstract

Subgame solving is a technique for scaling algorithms to large games by locally refining a precomputed blueprint strategy during gameplay. While straightforward in perfect-information games where search starts from the current state, subgame solving in imperfect-information games must account for hidden states and uncertainty about the opponent's past strategy. Gadget games were developed to ensure that the improved subgame strategy is robust against any possible opponent's strategy in a zero-sum game. Gadget games typically contain infinitely many Nash equilibria. We demonstrate that while these equilibria are equivalent in the gadget game, they yield vastly different performance in the full game, even when facing a rational opponent. We propose gadget game sequential equilibria as the preferred solution concept. We introduce modifications to the sequence-form linear program and counterfactual regret minimization that converge to these refined solutions with only mild additional computational cost. Additionally, we provide several new insights into the surprising superiority of the resolving gadget game over the max-margin gadget game. Our experiments compare different Nash equilibria of gadget games in several standard benchmark games, showing that our refined equilibria consistently outperform unrefined Nash equilibria, and can reduce the exploitability of the overall strategy by more than 50%