PIMAEX: Multi-Agent Exploration through Peer Incentivization

TL;DR

This paper introduces PIMAEX, a peer-incentivized reward mechanism for multi-agent reinforcement learning that enhances exploration by encouraging agents to influence each other, demonstrated to outperform non-incentivized methods in complex environments.

Contribution

The work proposes a novel peer-incentivized reward function for multi-agent exploration and integrates it with a communication-based training algorithm, advancing exploration strategies in multi-agent RL.

Findings

Agents with PIMAEX outperform others in complex environments.

Peer influence increases exploration efficiency.

Communication enhances the effectiveness of PIMAEX.

Abstract

While exploration in single-agent reinforcement learning has been studied extensively in recent years, considerably less work has focused on its counterpart in multi-agent reinforcement learning. To address this issue, this work proposes a peer-incentivized reward function inspired by previous research on intrinsic curiosity and influence-based rewards. The \textit{PIMAEX} reward, short for Peer-Incentivized Multi-Agent Exploration, aims to improve exploration in the multi-agent setting by encouraging agents to exert influence over each other to increase the likelihood of encountering novel states. We evaluate the \textit{PIMAEX} reward in conjunction with \textit{PIMAEX-Communication}, a multi-agent training algorithm that employs a communication channel for agents to influence one another. The evaluation is conducted in the \textit{Consume/Explore} environment, a partially observable environment with deceptive rewards, specifically designed to challenge the exploration vs.\ exploitation dilemma and the credit-assignment problem. The results empirically demonstrate that agents using the \textit{PIMAEX} reward with \textit{PIMAEX-Communication} outperform those that do not.