Replay For Safety

TL;DR

This paper investigates how experience replay sampling schemes affect the convergence and properties of policies in reinforcement learning, proposing biased replay as a means to achieve safer policies.

Contribution

It establishes conditions for convergence in tabular Q-learning and introduces biased replay as a method to control and improve policy safety.

Findings

Conditions for convergence with replay sampling schemes

Biased replay can modify policy properties

Potential for safer policies through replay biasing

Abstract

Experience replay \citep{lin1993reinforcement, mnih2015human} is a widely used technique to achieve efficient use of data and improved performance in RL algorithms. In experience replay, past transitions are stored in a memory buffer and re-used during learning. Various suggestions for sampling schemes from the replay buffer have been suggested in previous works, attempting to optimally choose those experiences which will most contribute to the convergence to an optimal policy. Here, we give some conditions on the replay sampling scheme that will ensure convergence, focusing on the well-known Q-learning algorithm in the tabular setting. After establishing sufficient conditions for convergence, we turn to suggest a slightly different usage for experience replay - replaying memories in a biased manner as a means to change the properties of the resulting policy. We initiate a rigorous study of experience replay as a tool to control and modify the properties of the resulting policy. In particular, we show that using an appropriate biased sampling scheme can allow us to achieve a \emph{safe} policy. We believe that using experience replay as a biasing mechanism that allows controlling the resulting policy in desirable ways is an idea with promising potential for many applications.