Shared Experience Actor-Critic for Multi-Agent Reinforcement Learning

TL;DR

This paper introduces SEAC, a shared experience actor-critic method that enhances exploration in multi-agent reinforcement learning, especially in sparse reward settings, leading to faster learning and higher returns.

Contribution

The paper presents a novel experience sharing approach within an actor-critic framework for multi-agent RL, improving exploration efficiency in sparse reward environments.

Findings

SEAC outperforms baselines and state-of-the-art algorithms in multiple environments.

Experience sharing accelerates learning and achieves higher returns.

In difficult tasks, sharing experience can be the difference between success and failure.

Abstract

Exploration in multi-agent reinforcement learning is a challenging problem, especially in environments with sparse rewards. We propose a general method for efficient exploration by sharing experience amongst agents. Our proposed algorithm, called Shared Experience Actor-Critic (SEAC), applies experience sharing in an actor-critic framework. We evaluate SEAC in a collection of sparse-reward multi-agent environments and find that it consistently outperforms two baselines and two state-of-the-art algorithms by learning in fewer steps and converging to higher returns. In some harder environments, experience sharing makes the difference between learning to solve the task and not learning at all.