Experience Replay for Continual Learning

TL;DR

This paper investigates the use of experience replay buffers to mitigate catastrophic forgetting in continual reinforcement learning without explicit task boundary signals, demonstrating effectiveness in Atari and DMLab environments.

Contribution

It introduces a simple, general approach using experience replay buffers for continual learning that performs well without requiring task labels or boundaries.

Findings

Replay buffers reduce catastrophic forgetting effectively.

Limited buffer size with random data discarding performs nearly as well as unlimited buffers.

Method matches performance of task-aware approaches in reinforcement learning environments.

Abstract

Continual learning is the problem of learning new tasks or knowledge while protecting old knowledge and ideally generalizing from old experience to learn new tasks faster. Neural networks trained by stochastic gradient descent often degrade on old tasks when trained successively on new tasks with different data distributions. This phenomenon, referred to as catastrophic forgetting, is considered a major hurdle to learning with non-stationary data or sequences of new tasks, and prevents networks from continually accumulating knowledge and skills. We examine this issue in the context of reinforcement learning, in a setting where an agent is exposed to tasks in a sequence. Unlike most other work, we do not provide an explicit indication to the model of task boundaries, which is the most general circumstance for a learning agent exposed to continuous experience. While various methods to counteract catastrophic forgetting have recently been proposed, we explore a straightforward, general, and seemingly overlooked solution - that of using experience replay buffers for all past events - with a mixture of on- and off-policy learning, leveraging behavioral cloning. We show that this strategy can still learn new tasks quickly yet can substantially reduce catastrophic forgetting in both Atari and DMLab domains, even matching the performance of methods that require task identities. When buffer storage is constrained, we confirm that a simple mechanism for randomly discarding data allows a limited size buffer to perform almost as well as an unbounded one.