Dynamics of Multi-Agent Actor-Critic Learning in Stochastic Games: from Multistability and Chaos to Stable Cooperation

TL;DR

This paper analyzes the dynamics of actor-critic multi-agent reinforcement learning in stochastic games, revealing how entropy regularization influences stability, chaos, and cooperation, with implications for designing robust multi-agent systems.

Contribution

It provides a dynamical systems analysis of MARL, showing how entropy regularization affects stability and cooperation in stochastic games, connecting EGT strategies to MARL dynamics.

Findings

Chaos in Matching Pennies is mitigated by entropy regularization.

Multistability in Prisoner's Dilemma includes cooperation and defection equilibria.

Entropy regularization enlarges the basin of attraction for cooperative states.

Abstract

Achieving robust coordination and cooperation is a central challenge in multi-agent reinforcement learning (MARL). Uncovering the mechanisms underlying such emergent behaviors calls for a dynamical understanding of learn processes. In this work, we investigate the dynamics of actor-critic agents in stochastic games, focusing on the impact of entropy regularization. By leveraging time-scale separation, we derive the system's evolution equations, which are then formally analyzed using dynamical systems theory. We find that in the constant-sum game of Matching Pennies, the system exhibits chaotic behavior. Entropy regularization mitigates this chaos and drives the dynamics toward convergence to fair cooperation. In contrast, in the general-sum game of the Prisoner's Dilemma, the system displays multistability. Interestingly, the three stable equilibria of the system correspond to the well-known ALLC (Always Cooperate), ALLD (Always Defect), and GRIM (Grim Trigger) strategies from evolutionary game theory (EGT). Entropy regularization strengthens system resilience by enlarging the basin of attraction of the cooperative equilibrium. Our findings reveal a close link between the mechanism of direct reciprocity in EGT and how cooperation emerges in MARL, offering insights for designing more robust and collaborative multi-agent systems.