Scalable spectral representations for multi-agent reinforcement learning in network MDPs

TL;DR

This paper introduces scalable spectral local representations for network MDPs in multi-agent reinforcement learning, enabling efficient learning by exploiting network dynamics and providing convergence guarantees.

Contribution

It derives scalable spectral local representations for network MDPs and develops a convergent algorithmic framework for continuous state-action spaces.

Findings

Effective in two benchmark problems

Outperforms generic function approximation methods

Provides theoretical convergence guarantees

Abstract

Network Markov Decision Processes (MDPs), a popular model for multi-agent control, pose a significant challenge to efficient learning due to the exponential growth of the global state-action space with the number of agents. In this work, utilizing the exponential decay property of network dynamics, we first derive scalable spectral local representations for network MDPs, which induces a network linear subspace for the local $Q$-function of each agent. Building on these local spectral representations, we design a scalable algorithmic framework for continuous state-action network MDPs, and provide end-to-end guarantees for the convergence of our algorithm. Empirically, we validate the effectiveness of our scalable representation-based approach on two benchmark problems, and demonstrate the advantages of our approach over generic function approximation approaches to representing the local $Q$-functions.