Provably Convergent Actor-Critic in Risk-averse MARL

TL;DR

This paper introduces a provably convergent Actor-Critic algorithm for risk-averse multi-agent reinforcement learning in Markov games, leveraging risk-averse equilibria to ensure convergence and practical applicability.

Contribution

It develops a novel two-timescale Actor-Critic method that converges globally in risk-averse Markov games, addressing a key challenge in multi-agent RL.

Findings

Proves global convergence with finite-sample guarantees.

Demonstrates superior convergence in empirical environments.

Validates effectiveness of risk-averse equilibria in MARL.

Abstract

Learning stationary policies in infinite-horizon general-sum Markov games (MGs) remains a fundamental open problem in Multi-Agent Reinforcement Learning (MARL). While stationary strategies are preferred for their practicality, computing stationary forms of classic game-theoretic equilibria is computationally intractable -- a stark contrast to the comparative ease of solving single-agent RL or zero-sum games. To bridge this gap, we study Risk-averse Quantal response Equilibria (RQE), a solution concept rooted in behavioral game theory that incorporates risk aversion and bounded rationality. We demonstrate that RQE possesses strong regularity conditions that make it uniquely amenable to learning in MGs. We propose a novel two-timescale Actor-Critic algorithm characterized by a fast-timescale actor and a slow-timescale critic. Leveraging the regularity of RQE, we prove that this approach achieves global convergence with finite-sample guarantees. We empirically validate our algorithm in several environments to demonstrate superior convergence properties compared to risk-neutral baselines.