Efficient Diversity-based Experience Replay for Deep Reinforcement Learning

TL;DR

This paper introduces EDER, a novel diversity-based experience replay method using determinantal point processes, which enhances learning efficiency in high-dimensional reinforcement learning tasks.

Contribution

The paper proposes EDER, a new experience replay technique employing diversity modeling and Cholesky decomposition to improve efficiency in complex environments.

Findings

Significantly improves learning efficiency in high-dimensional tasks

Achieves superior performance in robotic and indoor environment benchmarks

Effective sample selection via diversity enhances reinforcement learning

Abstract

Experience replay is widely used to improve learning efficiency in reinforcement learning by leveraging past experiences. However, existing experience replay methods, whether based on uniform or prioritized sampling, often suffer from low efficiency, particularly in real-world scenarios with high-dimensional state spaces. To address this limitation, we propose a novel approach, Efficient Diversity-based Experience Replay (EDER). EDER employs a determinantal point process to model the diversity between samples and prioritizes replay based on the diversity between samples. To further enhance learning efficiency, we incorporate Cholesky decomposition for handling large state spaces in realistic environments. Additionally, rejection sampling is applied to select samples with higher diversity, thereby improving overall learning efficacy. Extensive experiments are conducted on robotic manipulation tasks in MuJoCo, Atari games, and realistic indoor environments in Habitat. The results demonstrate that our approach not only significantly improves learning efficiency but also achieves superior performance in high-dimensional, realistic environments.