Deep (Predictive) Discounted Counterfactual Regret Minimization

TL;DR

This paper introduces a neural CFR algorithm that efficiently approximates advanced CFR variants, leading to faster convergence and stronger performance in imperfect-information games.

Contribution

It presents a novel model-free neural CFR method that overcomes limitations of existing approaches in integrating advanced CFR variants.

Findings

Faster convergence in typical imperfect-information games

Stronger adversarial performance in large poker game

Effective approximation of advanced CFR variants

Abstract

Counterfactual regret minimization (CFR) is a family of algorithms for effectively solving imperfect-information games. To enhance CFR's applicability in large games, researchers use neural networks to approximate its behavior. However, existing methods are mainly based on vanilla CFR and struggle to effectively integrate more advanced CFR variants. In this work, we propose an efficient model-free neural CFR algorithm, overcoming the limitations of existing methods in approximating advanced CFR variants. At each iteration, it collects variance-reduced sampled advantages based on a value network, fits cumulative advantages by bootstrapping, and applies discounting and clipping operations to simulate the update mechanisms of advanced CFR variants. Experimental results show that, compared with model-free neural algorithms, it exhibits faster convergence in typical imperfect-information games and demonstrates stronger adversarial performance in a large poker game.