Continual State Representation Learning for Reinforcement Learning using Generative Replay

TL;DR

This paper introduces a continual learning approach for reinforcement learning that uses variational auto-encoders and generative replay to maintain and update state representations efficiently without forgetting past knowledge.

Contribution

It presents a novel method combining variational auto-encoders with generative replay for continual state representation learning in reinforcement learning.

Findings

Effective in maintaining past knowledge during environment changes

Prevents catastrophic forgetting in incremental learning

Enables bounded system size with incremental updates

Abstract

We consider the problem of building a state representation model in a continual fashion. As the environment changes, the aim is to efficiently compress the sensory state's information without losing past knowledge. The learned features are then fed to a Reinforcement Learning algorithm to learn a policy. We propose to use Variational Auto-Encoders for state representation, and Generative Replay, i.e. the use of generated samples, to maintain past knowledge. We also provide a general and statistically sound method for automatic environment change detection. Our method provides efficient state representation as well as forward transfer, and avoids catastrophic forgetting. The resulting model is capable of incrementally learning information without using past data and with a bounded system size.