# Experience Replay Using Transition Sequences

## TL;DR

This paper introduces a method for selecting and constructing transition sequences for experience replay in reinforcement learning, improving sample efficiency and value function learning in off-policy settings.

## Contribution

It proposes a novel approach to replay sequence selection and construction, enhancing the effectiveness of experience replay in reinforcement learning.

## Key findings

- Improved value function learning on modified RL tasks
- Replayed sequences accelerate learning compared to standard methods
- Applicable to off-policy reinforcement learning scenarios

## Abstract

Experience replay is one of the most commonly used approaches to improve the sample efficiency of reinforcement learning algorithms. In this work, we propose an approach to select and replay sequences of transitions in order to accelerate the learning of a reinforcement learning agent in an off-policy setting. In addition to selecting appropriate sequences, we also artificially construct transition sequences using information gathered from previous agent-environment interactions. These sequences, when replayed, allow value function information to trickle down to larger sections of the state/state-action space, thereby making the most of the agent's experience. We demonstrate our approach on modified versions of standard reinforcement learning tasks such as the mountain car and puddle world problems and empirically show that it enables better learning of value functions as compared to other forms of experience replay. Further, we briefly discuss some of the possible extensions to this work, as well as applications and situations where this approach could be particularly useful.