Heterogeneous-Agent Reinforcement Learning

TL;DR

This paper introduces a new framework for cooperative multi-agent reinforcement learning with heterogeneous agents, providing algorithms with theoretical guarantees and demonstrating superior performance over existing methods.

Contribution

The paper proposes Heterogeneous-Agent Reinforcement Learning (HARL) algorithms, including HATRL, HATRPO, HAPPO, and the Heterogeneous-Agent Mirror Learning framework, with proven convergence and improved stability.

Findings

HARL algorithms outperform baselines like MAPPO and QMIX.

Theoretical guarantees include monotonic improvement and convergence to Nash Equilibrium.

Heterogeneous agents can be effectively coordinated with the proposed methods.

Abstract

The necessity for cooperation among intelligent machines has popularised cooperative multi-agent reinforcement learning (MARL) in AI research. However, many research endeavours heavily rely on parameter sharing among agents, which confines them to only homogeneous-agent setting and leads to training instability and lack of convergence guarantees. To achieve effective cooperation in the general heterogeneous-agent setting, we propose Heterogeneous-Agent Reinforcement Learning (HARL) algorithms that resolve the aforementioned issues. Central to our findings are the multi-agent advantage decomposition lemma and the sequential update scheme. Based on these, we develop the provably correct Heterogeneous-Agent Trust Region Learning (HATRL), and derive HATRPO and HAPPO by tractable approximations. Furthermore, we discover a novel framework named Heterogeneous-Agent Mirror Learning (HAML), which strengthens theoretical guarantees for HATRPO and HAPPO and provides a general template for cooperative MARL algorithmic designs. We prove that all algorithms derived from HAML inherently enjoy monotonic improvement of joint return and convergence to Nash Equilibrium. As its natural outcome, HAML validates more novel algorithms in addition to HATRPO and HAPPO, including HAA2C, HADDPG, and HATD3, which generally outperform their existing MA-counterparts. We comprehensively test HARL algorithms on six challenging benchmarks and demonstrate their superior effectiveness and stability for coordinating heterogeneous agents compared to strong baselines such as MAPPO and QMIX.