QTypeMix: Enhancing Multi-Agent Cooperative Strategies through Heterogeneous and Homogeneous Value Decomposition

TL;DR

QTypeMix is a novel multi-agent reinforcement learning method that improves cooperative strategies by separating homogeneous and heterogeneous value decomposition, utilizing attention mechanisms and hypernets to enhance learning without extra domain knowledge.

Contribution

It introduces a two-stage value decomposition approach with type feature extraction and advanced network structures, achieving state-of-the-art results in complex multi-agent tasks.

Findings

Achieves state-of-the-art performance on SMAC and SMACv2 maps.

Effectively learns deeper role features without extra domain knowledge.

Outperforms existing methods across various difficulty levels.

Abstract

In multi-agent cooperative tasks, the presence of heterogeneous agents is familiar. Compared to cooperation among homogeneous agents, collaboration requires considering the best-suited sub-tasks for each agent. However, the operation of multi-agent systems often involves a large amount of complex interaction information, making it more challenging to learn heterogeneous strategies. Related multi-agent reinforcement learning methods sometimes use grouping mechanisms to form smaller cooperative groups or leverage prior domain knowledge to learn strategies for different roles. In contrast, agents should learn deeper role features without relying on additional information. Therefore, we propose QTypeMix, which divides the value decomposition process into homogeneous and heterogeneous stages. QTypeMix learns to extract type features from local historical observations through the TE loss. In addition, we introduce advanced network structures containing attention mechanisms and hypernets to enhance the representation capability and achieve the value decomposition process. The results of testing the proposed method on 14 maps from SMAC and SMACv2 show that QTypeMix achieves state-of-the-art performance in tasks of varying difficulty.