CEM-RL: Combining evolutionary and gradient-based methods for policy search

TL;DR

This paper introduces CEM-RL, a novel method combining the cross-entropy method with TD3, an off-policy deep RL algorithm, to improve policy search by balancing sample efficiency and stability.

Contribution

It proposes a new combination scheme using CEM and TD3, demonstrating advantages over existing methods in benchmark evaluations.

Findings

CEM-RL achieves better performance than some existing methods.

CEM-RL offers a favorable trade-off between sample efficiency and stability.

The method performs well across standard deep RL benchmarks.

Abstract

Deep neuroevolution and deep reinforcement learning (deep RL) algorithms are two popular approaches to policy search. The former is widely applicable and rather stable, but suffers from low sample efficiency. By contrast, the latter is more sample efficient, but the most sample efficient variants are also rather unstable and highly sensitive to hyper-parameter setting. So far, these families of methods have mostly been compared as competing tools. However, an emerging approach consists in combining them so as to get the best of both worlds. Two previously existing combinations use either an ad hoc evolutionary algorithm or a goal exploration process together with the Deep Deterministic Policy Gradient (DDPG) algorithm, a sample efficient off-policy deep RL algorithm. In this paper, we propose a different combination scheme using the simple cross-entropy method (CEM) and Twin Delayed Deep Deterministic policy gradient (td3), another off-policy deep RL algorithm which improves over ddpg. We evaluate the resulting method, cem-rl, on a set of benchmarks classically used in deep RL. We show that cem-rl benefits from several advantages over its competitors and offers a satisfactory trade-off between performance and sample efficiency.