Double Prioritized State Recycled Experience Replay

TL;DR

This paper introduces Double-Prioritized State Recycled (DPSR) experience replay, a novel method that enhances reinforcement learning by prioritizing experiences during training and storage, leading to improved performance on Atari games.

Contribution

The paper proposes DPSR, which combines dual prioritization and state recycling in experience replay, significantly improving reinforcement learning outcomes over existing methods.

Findings

Achieved state-of-the-art results on many Atari games.

Outperformed original and prioritized experience replay methods.

Enhanced experience utilization through state recycling.

Abstract

Experience replay enables online reinforcement learning agents to store and reuse the previous experiences of interacting with the environment. In the original method, the experiences are sampled and replayed uniformly at random. A prior work called prioritized experience replay was developed where experiences are prioritized, so as to replay experiences seeming to be more important more frequently. In this paper, we develop a method called double-prioritized state-recycled (DPSR) experience replay, prioritizing the experiences in both training stage and storing stage, as well as replacing the experiences in the memory with state recycling to make the best of experiences that seem to have low priorities temporarily. We used this method in Deep Q-Networks (DQN), and achieved a state-of-the-art result, outperforming the original method and prioritized experience replay on many Atari games.