Finding and Certifying (Near-)Optimal Strategies in Black-Box Extensive-Form Games

TL;DR

This paper develops methods to find and certify near-optimal strategies in large, complex black-box extensive-form games using only game-play, without full game tree expansion, providing exploitability guarantees and convergence guarantees.

Contribution

It introduces a new approach that relaxes previous assumptions, enabling certification and equilibrium-finding using only game-play and regret minimization.

Findings

Achieves high-probability exploitability certificates with minimal game tree expansion.

Provides an equilibrium-finding algorithm with $ ilde O(1/ oot 2 T)$ convergence rate.

Demonstrates practical effectiveness in black-box settings through experiments.

Abstract

Often -- for example in war games, strategy video games, and financial simulations -- the game is given to us only as a black-box simulator in which we can play it. In these settings, since the game may have unknown nature action distributions (from which we can only obtain samples) and/or be too large to expand fully, it can be difficult to compute strategies with guarantees on exploitability. Recent work \cite{Zhang20:Small} resulted in a notion of certificate for extensive-form games that allows exploitability guarantees while not expanding the full game tree. However, that work assumed that the black box could sample or expand arbitrary nodes of the game tree at any time, and that a series of exact game solves (via, for example, linear programming) can be conducted to compute the certificate. Each of those two assumptions severely restricts the practical applicability of that method. In this work, we relax both of the assumptions. We show that high-probability certificates can be obtained with a black box that can do nothing more than play through games, using only a regret minimizer as a subroutine. As a bonus, we obtain an equilibrium-finding algorithm with $\tilde O(1/\sqrt{T})$ convergence rate in the extensive-form game setting that does not rely on a sampling strategy with lower-bounded reach probabilities (which MCCFR assumes). We demonstrate experimentally that, in the black-box setting, our methods are able to provide nontrivial exploitability guarantees while expanding only a small fraction of the game tree.