Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learning

TL;DR

This paper critically examines the implementation details and the role of the monotonicity constraint in QMIX-based multi-agent reinforcement learning, revealing that code optimizations and the constraint itself significantly impact performance and sample efficiency.

Contribution

It provides a detailed analysis of code-level optimizations and the effects of the monotonicity constraint in QMIX, challenging common assumptions and offering theoretical insights.

Findings

Normalized code optimizations improve QMIX performance in SMAC.

Monotonicity constraint enhances sample efficiency in cooperative tasks.

Code-level factors significantly influence the effectiveness of MARL algorithms.

Abstract

Many complex multi-agent systems such as robot swarms control and autonomous vehicle coordination can be modeled as Multi-Agent Reinforcement Learning (MARL) tasks. QMIX, a widely popular MARL algorithm, has been used as a baseline for the benchmark environments, e.g., Starcraft Multi-Agent Challenge (SMAC), Difficulty-Enhanced Predator-Prey (DEPP). Recent variants of QMIX target relaxing the monotonicity constraint of QMIX, allowing for performance improvement in SMAC. In this paper, we investigate the code-level optimizations of these variants and the monotonicity constraint. (1) We find that such improvements of the variants are significantly affected by various code-level optimizations. (2) The experiment results show that QMIX with normalized optimizations outperforms other works in SMAC; (3) beyond the common wisdom from these works, the monotonicity constraint can improve sample efficiency in SMAC and DEPP. We also discuss why monotonicity constraints work well in purely cooperative tasks with a theoretical analysis. We open-source the code at \url{https://github.com/hijkzzz/pymarl2}.