Distributionally Robust Cooperative Multi-Agent Reinforcement Learning via Robust Value Factorization

TL;DR

This paper introduces a distributionally robust value factorization framework for cooperative multi-agent reinforcement learning, enhancing robustness to environmental uncertainties and improving out-of-distribution performance.

Contribution

It proposes the DrIGM principle and develops robust variants of existing value-factorization architectures that ensure robustness and scalability in real-world uncertain environments.

Findings

Improved out-of-distribution performance on SustainGym and StarCraft environments.

Robust value factorization methods compatible with existing architectures.

Theoretical guarantees for robustness in multi-agent systems.

Abstract

Cooperative multi-agent reinforcement learning (MARL) commonly adopts centralized training with decentralized execution, where value-factorization methods enforce the individual-global-maximum (IGM) principle so that decentralized greedy actions recover the team-optimal joint action. However, the reliability of this recipe in real-world settings remains unreliable due to environmental uncertainties arising from the sim-to-real gap, model mismatch, and system noise. We address this gap by introducing Distributionally robust IGM (DrIGM), a principle that requires each agent's robust greedy action to align with the robust team-optimal joint action. We show that DrIGM holds for a novel definition of robust individual action values, which is compatible with decentralized greedy execution and yields a provable robustness guarantee for the whole system. Building on this foundation, we derive DrIGM-compliant robust variants of existing value-factorization architectures (e.g., VDN/QMIX/QTRAN) that (i) train on robust Q-targets, (ii) preserve scalability, and (iii) integrate seamlessly with existing codebases without bespoke per-agent reward shaping. Empirically, on high-fidelity SustainGym simulators and a StarCraft game environment, our methods consistently improve out-of-distribution performance. Code and data are available at https://github.com/crqu/robust-coMARL.