Improving Experience Replay through Modeling of Similar Transitions' Sets

TL;DR

This paper introduces COMPER, a reinforcement learning method that enhances experience replay by modeling similar transition sets, achieving competitive results with significantly fewer training frames.

Contribution

The paper presents COMPER, a novel experience replay approach using recurrence over similar transition sets, reducing training data requirements in reinforcement learning.

Findings

COMPER achieves comparable performance with fewer frames.

It outperforms baseline DQN on several Atari games.

Requires only 100,000 frames for effective training.

Abstract

In this work, we propose and evaluate a new reinforcement learning method, COMPact Experience Replay (COMPER), which uses temporal difference learning with predicted target values based on recurrence over sets of similar transitions, and a new approach for experience replay based on two transitions memories. Our objective is to reduce the required number of experiences to agent training regarding the total accumulated rewarding in the long run. Its relevance to reinforcement learning is related to the small number of observations that it needs to achieve results similar to that obtained by relevant methods in the literature, that generally demand millions of video frames to train an agent on the Atari 2600 games. We report detailed results from five training trials of COMPER for just 100,000 frames and about 25,000 iterations with a small experiences memory on eight challenging games of Arcade Learning Environment (ALE). We also present results for a DQN agent with the same experimental protocol on the same games set as the baseline. To verify the performance of COMPER on approximating a good policy from a smaller number of observations, we also compare its results with that obtained from millions of frames presented on the benchmark of ALE.