MMD-MIX: Value Function Factorisation with Maximum Mean Discrepancy for Cooperative Multi-Agent Reinforcement Learning

TL;DR

MMD-MIX introduces a novel approach combining distributional reinforcement learning and value decomposition with randomness to improve cooperative multi-agent reinforcement learning performance.

Contribution

It proposes MMD-MIX, integrating distributional RL and value decomposition with randomness, enhancing data efficiency and performance in multi-agent tasks.

Findings

MMD-MIX outperforms prior methods in SMAC environment.

Incorporating randomness improves learning stability.

Distributional approach enhances value estimation accuracy.

Abstract

In the real world, many tasks require multiple agents to cooperate with each other under the condition of local observations. To solve such problems, many multi-agent reinforcement learning methods based on Centralized Training with Decentralized Execution have been proposed. One representative class of work is value decomposition, which decomposes the global joint Q-value $Q_\text{jt}$ into individual Q-values $Q_a$ to guide individuals' behaviors, e.g. VDN (Value-Decomposition Networks) and QMIX. However, these baselines often ignore the randomness in the situation. We propose MMD-MIX, a method that combines distributional reinforcement learning and value decomposition to alleviate the above weaknesses. Besides, to improve data sampling efficiency, we were inspired by REM (Random Ensemble Mixture) which is a robust RL algorithm to explicitly introduce randomness into the MMD-MIX. The experiments demonstrate that MMD-MIX outperforms prior baselines in the StarCraft Multi-Agent Challenge (SMAC) environment.