Faster Game Solving via Predictive Blackwell Approachability: Connecting Regret Matching and Mirror Descent

TL;DR

This paper introduces predictive Blackwell approachability, connecting regret minimization algorithms to improve convergence speed in solving large-scale zero-sum games, significantly outperforming prior methods.

Contribution

It establishes a novel connection between regret matching algorithms and regret minimizers like FTRL and OMD, leading to faster game-solving algorithms.

Findings

Predictive RM+ converges much faster than prior algorithms in benchmark games.

The approach achieves up to two orders of magnitude speedup in convergence.

Experimental results validate the effectiveness across multiple zero-sum game benchmarks.

Abstract

Blackwell approachability is a framework for reasoning about repeated games with vector-valued payoffs. We introduce predictive Blackwell approachability, where an estimate of the next payoff vector is given, and the decision maker tries to achieve better performance based on the accuracy of that estimator. In order to derive algorithms that achieve predictive Blackwell approachability, we start by showing a powerful connection between four well-known algorithms. Follow-the-regularized-leader (FTRL) and online mirror descent (OMD) are the most prevalent regret minimizers in online convex optimization. In spite of this prevalence, the regret matching (RM) and regret matching+ (RM+) algorithms have been preferred in the practice of solving large-scale games (as the local regret minimizers within the counterfactual regret minimization framework). We show that RM and RM+ are the algorithms that result from running FTRL and OMD, respectively, to select the halfspace to force at all times in the underlying Blackwell approachability game. By applying the predictive variants of FTRL or OMD to this connection, we obtain predictive Blackwell approachability algorithms, as well as predictive variants of RM and RM+. In experiments across 18 common zero-sum extensive-form benchmark games, we show that predictive RM+ coupled with counterfactual regret minimization converges vastly faster than the fastest prior algorithms (CFR+, DCFR, LCFR) across all games but two of the poker games, sometimes by two or more orders of magnitude.