A Recipe for Stable Offline Multi-agent Reinforcement Learning

TL;DR

This paper identifies instability issues in offline multi-agent reinforcement learning due to non-linear value decomposition and proposes a normalization technique, SVN, to stabilize training and improve performance.

Contribution

The paper analyzes the causes of instability in offline MARL and introduces scale-invariant value normalization (SVN) to stabilize non-linear value decomposition.

Findings

SVN stabilizes actor-critic training in offline MARL.

Non-linear value decomposition causes value-scale amplification and instability.

The proposed recipe enhances offline MARL performance.

Abstract

Despite remarkable achievements in single-agent offline reinforcement learning (RL), multi-agent RL (MARL) has struggled to adopt this paradigm, largely persisting with on-policy training and self-play from scratch. One reason for this gap comes from the instability of non-linear value decomposition, leading prior works to avoid complex mixing networks in favor of linear value decomposition (e.g., VDN) with value regularization used in single-agent setups. In this work, we analyze the source of instability in non-linear value decomposition within the offline MARL setting. Our observations confirm that they induce value-scale amplification and unstable optimization. To alleviate this, we propose a simple technique, scale-invariant value normalization (SVN), that stabilizes actor-critic training without altering the Bellman fixed point. Empirically, we examine the interaction among key components of offline MARL (e.g., value decomposition, value learning, and policy extraction) and derive a practical recipe that unlocks its full potential.