Teaching an Old Dynamics New Tricks: Regularization-free Last-iterate Convergence in Zero-sum Games via BNN Dynamics

TL;DR

This paper introduces a regularization-free method using Brown-von Neumann-Nash dynamics for last-iterate convergence in zero-sum games, applicable to both normal-form and extensive-form games with neural approximation, outperforming existing approaches.

Contribution

It repurposes classical BNN dynamics for zero-sum games, providing convergence guarantees without regularization and developing a scalable neural implementation for complex game settings.

Findings

Achieves last-iterate convergence in noisy normal-form games.

Outperforms regularization-based methods in empirical tests.

Adapts quickly to nonstationarities in game environments.

Abstract

Zero-sum games are a fundamental setting for adversarial training and decision-making in multi-agent learning (MAL). Existing methods often ensure convergence to (approximate) Nash equilibria by introducing a form of regularization. Yet, regularization requires additional hyperparameters, which must be carefully tuned--a challenging task when the payoff structure is known, and considerably harder when the structure is unknown or subject to change. Motivated by this problem, we repurpose a classical model in evolutionary game theory, i.e., the Brown-von Neumann-Nash (BNN) dynamics, by leveraging the intrinsic convergence of this dynamics in zero-sum games without regularization, and provide last-iterate convergence guarantees in noisy normal-form games (NFGs). Importantly, to make this approach more applicable, we develop a novel framework with theoretical guarantees that integrates the BNN dynamics in extensive-form games (EFGs) through counterfactual weighting. Furthermore, we implement an algorithm that instantiates our framework with neural function approximation, enabling scalable learning in both NFGs and EFGs. Empirical results show that our method quickly adapts to nonstationarities, outperforming the state-of-the-art regularization-based approach.