GEP-PG: Decoupling Exploration and Exploitation in Deep Reinforcement Learning Algorithms

TL;DR

GEP-PG introduces a hybrid reinforcement learning approach that combines exploration-focused methods with gradient-based exploitation to improve learning efficiency in challenging continuous control tasks.

Contribution

This paper presents GEP-PG, a novel method that sequentially combines Goal Exploration Processes with DDPG, enhancing performance in deceptive and complex environments.

Findings

GEP-PG outperforms DDPG in deceptive reward environments.

Sequential combination improves learning efficiency and robustness.

Method is effective on both low-dimensional and large-scale benchmarks.

Abstract

In continuous action domains, standard deep reinforcement learning algorithms like DDPG suffer from inefficient exploration when facing sparse or deceptive reward problems. Conversely, evolutionary and developmental methods focusing on exploration like Novelty Search, Quality-Diversity or Goal Exploration Processes explore more robustly but are less efficient at fine-tuning policies using gradient descent. In this paper, we present the GEP-PG approach, taking the best of both worlds by sequentially combining a Goal Exploration Process and two variants of DDPG. We study the learning performance of these components and their combination on a low dimensional deceptive reward problem and on the larger Half-Cheetah benchmark. We show that DDPG fails on the former and that GEP-PG improves over the best DDPG variant in both environments. Supplementary videos and discussion can be found at http://frama.link/gep_pg, the code at http://github.com/flowersteam/geppg.