Maximum Correntropy Value Decomposition for Multi-agent Deep Reinforcemen Learning

TL;DR

This paper introduces MCVD, a novel value decomposition method for multi-agent deep reinforcement learning that adaptively handles non-monotonic value functions, overcoming limitations of fixed-weight schemes like Weighted QMIX.

Contribution

The paper proposes MCVD, a new algorithm based on maximum correntropy, to improve value decomposition in multi-agent reinforcement learning, especially for non-monotonic problems.

Findings

MCVD effectively handles non-monotonic value decomposition problems.

MCVD demonstrates broad applicability and stability across multiple scenarios.

MCVD simplifies implementation compared to existing methods.

Abstract

We explore value decomposition solutions for multi-agent deep reinforcement learning in the popular paradigm of centralized training with decentralized execution(CTDE). As the recognized best solution to CTDE, Weighted QMIX is cutting-edge on StarCraft Multi-agent Challenge (SMAC), with a weighting scheme implemented on QMIX to place more emphasis on the optimal joint actions. However, the fixed weight requires manual tuning according to the application scenarios, which painfully prevents Weighted QMIX from being used in broader engineering applications. In this paper, we first demonstrate the flaw of Weighted QMIX using an ordinary One-Step Matrix Game (OMG), that no matter how the weight is chosen, Weighted QMIX struggles to deal with non-monotonic value decomposition problems with a large variance of reward distributions. Then we characterize the problem of value decomposition as an Underfitting One-edged Robust Regression problem and make the first attempt to give a solution to the value decomposition problem from the perspective of information-theoretical learning. We introduce the Maximum Correntropy Criterion (MCC) as a cost function to dynamically adapt the weight to eliminate the effects of minimum in reward distributions. We simplify the implementation and propose a new algorithm called MCVD. A preliminary experiment conducted on OMG shows that MCVD could deal with non-monotonic value decomposition problems with a large tolerance of kernel bandwidth selection. Further experiments are carried out on Cooperative-Navigation and multiple SMAC scenarios, where MCVD exhibits unprecedented ease of implementation, broad applicability, and stability.