AIIR-MIX: Multi-Agent Reinforcement Learning Meets Attention Individual Intrinsic Reward Mixing Network

TL;DR

AIIR-MIX introduces a novel attention-based intrinsic reward network and a dynamic mixing approach for multi-agent reinforcement learning, significantly improving cooperation and performance in complex environments like StarCraft II.

Contribution

The paper presents a new intrinsic reward network using attention mechanisms and a non-linear, dynamic reward mixing method for MARL, enhancing agent cooperation and performance.

Findings

AIIR-MIX outperforms state-of-the-art MARL methods in StarCraft II battle games.

The method dynamically assigns intrinsic rewards based on agent contributions.

Ablation studies confirm the effectiveness of the reward mixing strategy.

Abstract

Deducing the contribution of each agent and assigning the corresponding reward to them is a crucial problem in cooperative Multi-Agent Reinforcement Learning (MARL). Previous studies try to resolve the issue through designing an intrinsic reward function, but the intrinsic reward is simply combined with the environment reward by summation in these studies, which makes the performance of their MARL framework unsatisfactory. We propose a novel method named Attention Individual Intrinsic Reward Mixing Network (AIIR-MIX) in MARL, and the contributions of AIIR-MIX are listed as follows:(a) we construct a novel intrinsic reward network based on the attention mechanism to make teamwork more effective. (b) we propose a Mixing network that is able to combine intrinsic and extrinsic rewards non-linearly and dynamically in response to changing conditions of the environment. We compare AIIR-MIX with many State-Of-The-Art (SOTA) MARL methods on battle games in StarCraft II. And the results demonstrate that AIIR-MIX performs admirably and can defeat the current advanced methods on average test win rate. To validate the effectiveness of AIIR-MIX, we conduct additional ablation studies. The results show that AIIR-MIX can dynamically assign each agent a real-time intrinsic reward in accordance with their actual contribution.