ComaDICE: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularization

TL;DR

ComaDICE introduces a stationary distribution regularization method for offline cooperative multi-agent reinforcement learning, effectively addressing distributional shift and outperforming existing methods on benchmark tasks.

Contribution

The paper proposes a novel stationary distribution regularizer for offline MARL, along with a multi-agent value decomposition strategy, advancing the state-of-the-art in the field.

Findings

Superior performance on multi-agent MuJoCo benchmarks.

Outperforms existing offline MARL methods on StarCraft II tasks.

Effective handling of distributional shift in multi-agent settings.

Abstract

Offline reinforcement learning (RL) has garnered significant attention for its ability to learn effective policies from pre-collected datasets without the need for further environmental interactions. While promising results have been demonstrated in single-agent settings, offline multi-agent reinforcement learning (MARL) presents additional challenges due to the large joint state-action space and the complexity of multi-agent behaviors. A key issue in offline RL is the distributional shift, which arises when the target policy being optimized deviates from the behavior policy that generated the data. This problem is exacerbated in MARL due to the interdependence between agents' local policies and the expansive joint state-action space. Prior approaches have primarily addressed this challenge by incorporating regularization in the space of either Q-functions or policies. In this work, we introduce a regularizer in the space of stationary distributions to better handle distributional shift. Our algorithm, ComaDICE, offers a principled framework for offline cooperative MARL by incorporating stationary distribution regularization for the global learning policy, complemented by a carefully structured multi-agent value decomposition strategy to facilitate multi-agent training. Through extensive experiments on the multi-agent MuJoCo and StarCraft II benchmarks, we demonstrate that ComaDICE achieves superior performance compared to state-of-the-art offline MARL methods across nearly all tasks.