Memory Augmented Self-Play

TL;DR

This paper introduces a memory-augmented self-play framework that enhances exploration and performance of reinforcement learning agents by leveraging external memory to store past experiences, leading to more diverse tasks and faster learning.

Contribution

It proposes a novel memory-augmented self-play method that improves exploration efficiency and performance in reinforcement learning by integrating external memory into the self-play process.

Findings

Memory augmentation leads to more diverse self-play tasks.

Agents with memory outperform those without in exploration speed.

Pretrained agents in the memory-augmented setting perform better.

Abstract

Self-play is an unsupervised training procedure which enables the reinforcement learning agents to explore the environment without requiring any external rewards. We augment the self-play setting by providing an external memory where the agent can store experience from the previous tasks. This enables the agent to come up with more diverse self-play tasks resulting in faster exploration of the environment. The agent pretrained in the memory augmented self-play setting easily outperforms the agent pretrained in no-memory self-play setting.