ESCHER: Eschewing Importance Sampling in Games by Computing a History Value Function to Estimate Regret

TL;DR

ESCHER introduces a novel, unbiased neural method for approximating Nash equilibria in large games, significantly reducing variance and outperforming existing methods like DREAM and NFSP, especially in very large games.

Contribution

ESCHER provides a model-free, importance sampling-free neural approach that guarantees convergence and achieves superior performance in large-scale games.

Findings

ESCHER has significantly lower variance in regret estimation than DREAM.

ESCHER outperforms DREAM and NFSP in large games, including dark chess.

In dark chess, ESCHER beats DREAM and NFSP over 90% of the time.

Abstract

Recent techniques for approximating Nash equilibria in very large games leverage neural networks to learn approximately optimal policies (strategies). One promising line of research uses neural networks to approximate counterfactual regret minimization (CFR) or its modern variants. DREAM, the only current CFR-based neural method that is model free and therefore scalable to very large games, trains a neural network on an estimated regret target that can have extremely high variance due to an importance sampling term inherited from Monte Carlo CFR (MCCFR). In this paper we propose an unbiased model-free method that does not require any importance sampling. Our method, ESCHER, is principled and is guaranteed to converge to an approximate Nash equilibrium with high probability. We show that the variance of the estimated regret of ESCHER is orders of magnitude lower than DREAM and other baselines. We then show that ESCHER outperforms the prior state of the art -- DREAM and neural fictitious self play (NFSP) -- on a number of games and the difference becomes dramatic as game size increases. In the very large game of dark chess, ESCHER is able to beat DREAM and NFSP in a head-to-head competition over $90\%$ of the time.