MESA: Cooperative Meta-Exploration in Multi-Agent Learning through Exploiting State-Action Space Structure

TL;DR

MESA is a meta-exploration method that enhances cooperative multi-agent reinforcement learning by identifying high-reward state-action subspaces and learning diverse exploration policies, leading to improved performance especially in sparse-reward environments.

Contribution

MESA introduces a novel approach to exploration in MARL by learning to cover high-reward subspaces, improving efficiency and generalization in sparse-reward tasks.

Findings

MESA outperforms baseline methods in multi-agent particle and MuJoCo environments.

Learned exploration policies significantly improve success rates in sparse-reward tasks.

MESA generalizes well to more challenging test tasks.

Abstract

Multi-agent reinforcement learning (MARL) algorithms often struggle to find strategies close to Pareto optimal Nash Equilibrium, owing largely to the lack of efficient exploration. The problem is exacerbated in sparse-reward settings, caused by the larger variance exhibited in policy learning. This paper introduces MESA, a novel meta-exploration method for cooperative multi-agent learning. It learns to explore by first identifying the agents' high-rewarding joint state-action subspace from training tasks and then learning a set of diverse exploration policies to "cover" the subspace. These trained exploration policies can be integrated with any off-policy MARL algorithm for test-time tasks. We first showcase MESA's advantage in a multi-step matrix game. Furthermore, experiments show that with learned exploration policies, MESA achieves significantly better performance in sparse-reward tasks in several multi-agent particle environments and multi-agent MuJoCo environments, and exhibits the ability to generalize to more challenging tasks at test time.