Independent Learning of Nash Equilibria in Partially Observable Markov Potential Games with Decoupled Dynamics

TL;DR

This paper introduces an independent learning algorithm for Nash equilibria in a subclass of partially observable Markov games with decoupled dynamics, achieving near-polynomial complexity without communication.

Contribution

It proposes a novel independent learning method for POMGs with independent state transitions, enabling convergence to approximate Nash equilibria without requiring centralized information.

Findings

Converges to approximate Nash equilibria in POMGs with decoupled dynamics.

Achieves quasi-polynomial sample and computational complexity.

Uses finite history windows under a filter stability assumption.

Abstract

We study Nash equilibrium learning in partially observable Markov games (POMGs), a multi-agent reinforcement learning framework in which agents cannot fully observe the underlying state. Prior work in this setting relies on centralization or information sharing, and suffers from sample and computational complexity that scales exponentially in the number of players. We focus on a subclass of POMGs with independent state transitions, where agents remain coupled through their rewards, and assume that the underlying fully observed Markov game is a Markov potential game. For this class, we present an independent learning algorithm in which players, observing only their own actions and observations and without communication, jointly converge to an approximate Nash equilibrium. Due to partial observability, optimal policies may in general depend on the full action-observation history. Under a filter stability assumption, we show that policies based on finite history windows provide sufficient approximation guarantees. This enables us to approximate the POMG by a surrogate Markov game that is near-potential, leading to quasi-polynomial sample and computational complexity for independent Nash equilibrium learning in the underlying POMG.