Three-Player Game Training Dynamics

TL;DR

This paper investigates the dynamics of three-player game training, revealing conditions for convergence, the impact of update order, and demonstrating that a novel maximizer-first update order can lead to convergence on Nash equilibria.

Contribution

It introduces a new update order for three-player games, maximizer-first, which can achieve convergence on Nash equilibria, unlike traditional update schemes.

Findings

Three-player games often do not converge to Nash equilibrium.

Maximizer-first update order enables convergence to Nash equilibrium.

Maximizer-first updates outperform other orders with various momentum settings.

Abstract

This work explores three-player game training dynamics, under what conditions three-player games converge and the equilibria the converge on. In contrast to prior work, we examine a three-player game architecture in which all players explicitly interact with each other. Prior work analyzes games in which two of three agents interact with only one other player, constituting dual two-player games. We explore three-player game training dynamics using an extended version of a simplified bilinear smooth game, called a simplified trilinear smooth game. We find that trilinear games do not converge on the Nash equilibrium in most cases, rather converging on a fixed point which is optimal for two players, but not for the third. Further, we explore how the order of the updates influences convergence. In addition to alternating and simultaneous updates, we explore a new update order--maximizer-first--which is only possible in a three-player game. We find that three-player games can converge on a Nash equilibrium using maximizer-first updates. Finally, we experiment with differing momentum values for each player in a trilinear smooth game under all three update orders and show that maximizer-first updates achieve more optimal results in a larger set of player-specific momentum value triads than other update orders.