Equilibrium Selection for Multi-agent Reinforcement Learning: A Unified Framework

TL;DR

This paper introduces a unified actor-critic framework for multi-agent reinforcement learning that guides the selection of equilibria with desirable properties like high social welfare, extending equilibrium selection principles from normal-form games to stochastic games.

Contribution

It proposes a novel actor-critic method that incorporates equilibrium selection rules, ensuring convergence to equilibria with optimal social welfare in stochastic games.

Findings

The framework achieves potential-maximizing policies in Markov potential games.

It finds Pareto-optimal equilibria in general-sum stochastic games.

Sample-based implementation demonstrates practical applicability.

Abstract

While multi-agent reinforcement learning (MARL) has produced numerous algorithms that converge to Nash or related equilibria, such equilibria are often non-unique and can exhibit widely varying efficiency. This raises a fundamental question: how can one design learning dynamics that not only converge to equilibrium but also select equilibria with desirable performance, such as high social welfare? In contrast to the MARL literature, equilibrium selection has been extensively studied in normal-form games, where decentralized dynamics are known to converge to potential-maximizing or Pareto-optimal Nash equilibria (NEs). Motivated by these results, we study equilibrium selection in finite-horizon stochastic games. We propose a unified actor-critic framework in which a critic learns state-action value functions, and an actor applies a classical equilibrium-selection rule state-wise, treating learned values as stage-game payoffs. We show that, under standard stochastic stability assumptions, the stochastically stable policies of the resulting dynamics inherit the equilibrium selection properties of the underlying normal-form learning rule. As consequences, we obtain potential-maximizing policies in Markov potential games and Pareto-optimal (Markov perfect) equilibria in general-sum stochastic games, together with sample-based implementation of the framework.