SMIX($\lambda$): Enhancing Centralized Value Functions for Cooperative Multi-Agent Reinforcement Learning

TL;DR

SMIX(λ) introduces an off-policy training method for centralized value functions in multi-agent reinforcement learning, utilizing λ-return to improve stability and performance, demonstrated on the SMAC benchmark.

Contribution

It proposes SMIX(λ), a novel off-policy training approach that leverages λ-return for stable, efficient learning of centralized value functions in MARL, connecting to Q(λ) theory.

Findings

Outperforms state-of-the-art MARL methods on SMAC benchmark

Enhances performance of CTDE algorithms by improving CVFs

Shares convergence properties with Q(λ) approach

Abstract

Learning a stable and generalizable centralized value function (CVF) is a crucial but challenging task in multi-agent reinforcement learning (MARL), as it has to deal with the issue that the joint action space increases exponentially with the number of agents in such scenarios. This paper proposes an approach, named SMIX(${\lambda}$), to address the issue using an efficient off-policy centralized training method within a flexible learner search space. As importance sampling for such off-policy training is both computationally costly and numerically unstable, we proposed to use the ${\lambda}$-return as a proxy to compute the TD error. With this new loss function objective, we adopt a modified QMIX network structure as the base to train our model. By further connecting it with the ${Q(\lambda)}$ approach from an unified expectation correction viewpoint, we show that the proposed SMIX(${\lambda}$) is equivalent to ${Q(\lambda)}$ and hence shares its convergence properties, while without being suffered from the aforementioned curse of dimensionality problem inherent in MARL. Experiments on the StarCraft Multi-Agent Challenge (SMAC) benchmark demonstrate that our approach not only outperforms several state-of-the-art MARL methods by a large margin, but also can be used as a general tool to improve the overall performance of other CTDE-type algorithms by enhancing their CVFs.