Integrating Episodic Memory into a Reinforcement Learning Agent using Reservoir Sampling

TL;DR

This paper introduces a novel reservoir sampling-based external memory mechanism for reinforcement learning agents, enabling efficient online learning and improved recall of useful past states for better decision-making.

Contribution

It presents a new algorithm that maintains a fixed-size external memory of past states using reservoir sampling, allowing online reinforcement learning to selectively remember useful information.

Findings

Enables online gradient computation for external memory updates.

Improves the agent's ability to recall relevant past states.

Supports efficient long-term dependency learning in reinforcement learning.

Abstract

Episodic memory is a psychology term which refers to the ability to recall specific events from the past. We suggest one advantage of this particular type of memory is the ability to easily assign credit to a specific state when remembered information is found to be useful. Inspired by this idea, and the increasing popularity of external memory mechanisms to handle long-term dependencies in deep learning systems, we propose a novel algorithm which uses a reservoir sampling procedure to maintain an external memory consisting of a fixed number of past states. The algorithm allows a deep reinforcement learning agent to learn online to preferentially remember those states which are found to be useful to recall later on. Critically this method allows for efficient online computation of gradient estimates with respect to the write process of the external memory. Thus unlike most prior mechanisms for external memory it is feasible to use in an online reinforcement learning setting.