Policy Abstraction and Nash Refinement in Tree-Exploiting PSRO

TL;DR

This paper advances Tree-exploiting PSRO by introducing scalable game tree representations and refined equilibrium strategies, enabling faster convergence in complex imperfect-information games.

Contribution

It proposes a scalable empirical game tree representation and a generalized backward induction algorithm for refined Nash equilibria in TE-PSRO.

Findings

TE-PSRO converges faster with SPE-based strategy generation.

The approach handles complex imperfect-information games efficiently.

Empirical results show reasonable time and memory usage.

Abstract

Policy Space Response Oracles (PSRO) interleaves empirical game-theoretic analysis with deep reinforcement learning (DRL) to solve games too complex for traditional analytic methods. Tree-exploiting PSRO (TE-PSRO) is a variant of this approach that iteratively builds a coarsened empirical game model in extensive form using data obtained from querying a simulator that represents a detailed description of the game. We make two main methodological advances to TE-PSRO that enhance its applicability to complex games of imperfect information. First, we introduce a scalable representation for the empirical game tree where edges correspond to implicit policies learned through DRL. These policies cover conditions in the underlying game abstracted in the game model, supporting sustainable growth of the tree over epochs. Second, we leverage extensive form in the empirical model by employing refined Nash equilibria to direct strategy exploration. To enable this, we give a modular and scalable algorithm based on generalized backward induction for computing a subgame perfect equilibrium (SPE) in an imperfect-information game. We experimentally evaluate our approach on a suite of games including an alternating-offer bargaining game with outside offers; our results demonstrate that TE-PSRO converges toward equilibrium faster when new strategies are generated based on SPE rather than Nash equilibrium, and with reasonable time/memory requirements for the growing empirical model.