Efficient Last-iterate Convergence Algorithms in Solving Games

TL;DR

This paper proves last-iterate convergence of CFR$^+$ in learning Nash equilibria in extensive-form games and introduces RTCFR$^+$, a new algorithm that outperforms existing methods with strong theoretical guarantees.

Contribution

The paper demonstrates that CFR$^+$ achieves last-iterate convergence in perturbed regularized EFGs and develops RTCFR$^+$, a new algorithm with improved empirical performance and stability.

Findings

RTCFR$^+$ significantly outperforms existing algorithms.

CFR$^+$ achieves last-iterate convergence in perturbed regularized EFGs.

Enhanced stability of CFR$^+$ is crucial for empirical convergence.

Abstract

To establish last-iterate convergence for Counterfactual Regret Minimization (CFR) algorithms in learning a Nash equilibrium (NE) of extensive-form games (EFGs), recent studies reformulate learning an NE of the original EFG as learning the NEs of a sequence of (perturbed) regularized EFGs. Consequently, proving last-iterate convergence in solving the original EFG reduces to proving last-iterate convergence in solving (perturbed) regularized EFGs. However, the empirical convergence rates of the algorithms in these studies are suboptimal, since they do not utilize Regret Matching (RM)-based CFR algorithms to solve perturbed EFGs, which are known the exceptionally fast empirical convergence rates. Additionally, since solving multiple perturbed regularized EFGs is required, fine-tuning across all such games is infeasible, making parameter-free algorithms highly desirable. In this paper, we prove that CFR$^+$, a classical parameter-free RM-based CFR algorithm, achieves last-iterate convergence in learning an NE of perturbed regularized EFGs. Leveraging CFR$^+$ to solve perturbed regularized EFGs, we get Reward Transformation CFR$^+$ (RTCFR$^+$). Importantly, we extend prior work on the parameter-free property of CFR$^+$, enhancing its stability, which is crucial for the empirical convergence of RTCFR$^+$. Experiments show that RTCFR$^+$ significantly outperforms existing algorithms with theoretical last-iterate convergence guarantees.