Multi-agent Exploration with Sub-state Entropy Estimation

TL;DR

This paper introduces MESE, a novel multi-agent exploration method that uses sub-state entropy estimation to promote cooperative exploration, significantly enhancing performance in complex multi-agent environments like StarCraft.

Contribution

MESE is a new exploration approach that incentivizes cooperation through entropy-based sub-state selection, easily integrated into existing MARL algorithms.

Findings

MESE improves MAPPO performance on SMAC tasks.

Entropy-based sub-state selection effectively guides cooperative exploration.

MESE is compatible with most MARL algorithms.

Abstract

Researchers have integrated exploration techniques into multi-agent reinforcement learning (MARL) algorithms, drawing on their remarkable success in deep reinforcement learning. Nonetheless, exploration in MARL presents a more substantial challenge, as agents need to coordinate their efforts in order to achieve comprehensive state coverage. Reaching a unanimous agreement on which kinds of states warrant exploring can be a struggle for agents in this context. We introduce \textbf{M}ulti-agent \textbf{E}xploration based on \textbf{S}ub-state \textbf{E}ntropy (MESE) to address this limitation. This novel approach incentivizes agents to explore states cooperatively by directing them to achieve consensus via an extra team reward. Calculating the additional reward is based on the novelty of the current sub-state that merits cooperative exploration. MESE employs a conditioned entropy approach to select the sub-state, using particle-based entropy estimation to calculate the entropy. MESE is a plug-and-play module that can be seamlessly integrated into most existing MARL algorithms, which makes it a highly effective tool for reinforcement learning. Our experiments demonstrate that MESE can substantially improve the MAPPO's performance on various tasks in the StarCraft multi-agent challenge (SMAC).