Coordinated Exploration via Intrinsic Rewards for Multi-Agent Reinforcement Learning

TL;DR

This paper introduces a hierarchical framework for multi-agent reinforcement learning that uses coordinated intrinsic rewards and dynamic exploration strategies to improve performance in sparse reward environments.

Contribution

It proposes a novel hierarchical approach that enables multi-agent coordination through intrinsic rewards and adaptive exploration mode selection.

Findings

Outperforms state-of-the-art methods in cooperative sparse reward tasks.

Effectively adapts exploration strategies to different multi-stage tasks.

Enhances coordination among agents through learned intrinsic reward mechanisms.

Abstract

Solving tasks with sparse rewards is one of the most important challenges in reinforcement learning. In the single-agent setting, this challenge is addressed by introducing intrinsic rewards that motivate agents to explore unseen regions of their state spaces; however, applying these techniques naively to the multi-agent setting results in agents exploring independently, without any coordination among themselves. Exploration in cooperative multi-agent settings can be accelerated and improved if agents coordinate their exploration. In this paper we introduce a framework for designing intrinsic rewards which consider what other agents have explored such that the agents can coordinate. Then, we develop an approach for learning how to dynamically select between several exploration modalities to maximize extrinsic rewards. Concretely, we formulate the approach as a hierarchical policy where a high-level controller selects among sets of policies trained on diverse intrinsic rewards and the low-level controllers learn the action policies of all agents under these specific rewards. We demonstrate the effectiveness of the proposed approach in cooperative domains with sparse rewards where state-of-the-art methods fail and challenging multi-stage tasks that necessitate changing modes of coordination.