Individual Contributions as Intrinsic Exploration Scaffolds for Multi-agent Reinforcement Learning

TL;DR

This paper introduces ICES, a novel method for multi-agent reinforcement learning that uses individual contribution-based intrinsic rewards to improve exploration in sparse reward environments, leveraging global transition information during training.

Contribution

The paper proposes ICES, a new approach that assesses individual agent contributions to guide exploration, separating exploration and exploitation policies for better learning.

Findings

ICES outperforms baselines in cooperative tasks with sparse rewards.

The method effectively guides agents to impactful actions during training.

Experimental results on GRF and SMAC show improved exploration capabilities.

Abstract

In multi-agent reinforcement learning (MARL), effective exploration is critical, especially in sparse reward environments. Although introducing global intrinsic rewards can foster exploration in such settings, it often complicates credit assignment among agents. To address this difficulty, we propose Individual Contributions as intrinsic Exploration Scaffolds (ICES), a novel approach to motivate exploration by assessing each agent's contribution from a global view. In particular, ICES constructs exploration scaffolds with Bayesian surprise, leveraging global transition information during centralized training. These scaffolds, used only in training, help to guide individual agents towards actions that significantly impact the global latent state transitions. Additionally, ICES separates exploration policies from exploitation policies, enabling the former to utilize privileged global information during training. Extensive experiments on cooperative benchmark tasks with sparse rewards, including Google Research Football (GRF) and StarCraft Multi-agent Challenge (SMAC), demonstrate that ICES exhibits superior exploration capabilities compared with baselines. The code is publicly available at https://github.com/LXXXXR/ICES.