AIR: Unifying Individual and Collective Exploration in Cooperative Multi-Agent Reinforcement Learning

TL;DR

This paper introduces AIR, a novel adaptive exploration method for cooperative multi-agent reinforcement learning that unifies individual and collective exploration, improving training efficiency and effectiveness.

Contribution

AIR combines adversarial identity recognition with adaptive exploration, enabling simultaneous individual and collective exploration without additional structural complexity.

Findings

AIR enhances exploration efficiency in MARL tasks

AIR improves training effectiveness across various environments

Theoretical proof supports AIR's dual exploration capabilities

Abstract

Exploration in cooperative multi-agent reinforcement learning (MARL) remains challenging for value-based agents due to the absence of an explicit policy. Existing approaches include individual exploration based on uncertainty towards the system and collective exploration through behavioral diversity among agents. However, the introduction of additional structures often leads to reduced training efficiency and infeasible integration of these methods. In this paper, we propose Adaptive exploration via Identity Recognition~(AIR), which consists of two adversarial components: a classifier that recognizes agent identities from their trajectories, and an action selector that adaptively adjusts the mode and degree of exploration. We theoretically prove that AIR can facilitate both individual and collective exploration during training, and experiments also demonstrate the efficiency and effectiveness of AIR across various tasks.