ME-IGM: Individual-Global-Max in Maximum Entropy Multi-Agent Reinforcement Learning

TL;DR

This paper introduces ME-IGM, a maximum entropy multi-agent reinforcement learning algorithm that aligns local and global policies to improve exploration and credit assignment, demonstrating superior performance in complex cooperative tasks.

Contribution

The paper proposes a novel order-preserving transformation to address misalignment issues in maximum entropy MARL, enabling compatibility with any IGM-compliant credit assignment mechanism.

Findings

ME-IGM achieves state-of-the-art results in 17 scenarios.

Empirical evaluation confirms improved exploration and coordination.

Variants ME-QMIX and ME-QPLEX outperform existing methods.

Abstract

Multi-agent credit assignment is a fundamental challenge for cooperative multi-agent reinforcement learning (MARL), where a team of agents learn from shared reward signals. The Individual-Global-Max (IGM) condition is a widely used principle for multi-agent credit assignment, requiring that the joint action determined by individual Q-functions maximizes the global Q-value. Meanwhile, the principle of maximum entropy has been leveraged to enhance exploration in MARL. However, we identify a critical limitation in existing maximum entropy MARL methods: a misalignment arises between local policies and the joint policy that maximizes the global Q-value, leading to violations of the IGM condition. To address this misalignment, we propose an order-preserving transformation. Building on it, we introduce ME-IGM, a novel maximum entropy MARL algorithm compatible with any credit assignment mechanism that satisfies the IGM condition while enjoying the benefits of maximum entropy exploration. We empirically evaluate two variants of ME-IGM: ME-QMIX and ME-QPLEX, in non-monotonic matrix games, and demonstrate their state-of-the-art performance across 17 scenarios in SMAC-v2 and Overcooked.