Evolutionary Strategy Guided Reinforcement Learning via MultiBuffer Communication

TL;DR

This paper introduces a novel Evolutionary Reinforcement Learning framework that combines Evolutionary Strategies with TD3 using a multi-buffer system, enhancing policy search and performance on control tasks.

Contribution

It presents a new multi-buffer approach that improves policy exploration and performance in Evolutionary Reinforcement Learning by integrating Evolutionary Strategies with TD3.

Findings

Outperforms CEM-RL on 3 of 4 MuJoCo tasks

Enables freer policy search without buffer overpopulation issues

Demonstrates competitive results with state-of-the-art algorithms

Abstract

Evolutionary Algorithms and Deep Reinforcement Learning have both successfully solved control problems across a variety of domains. Recently, algorithms have been proposed which combine these two methods, aiming to leverage the strengths and mitigate the weaknesses of both approaches. In this paper we introduce a new Evolutionary Reinforcement Learning model which combines a particular family of Evolutionary algorithm called Evolutionary Strategies with the off-policy Deep Reinforcement Learning algorithm TD3. The framework utilises a multi-buffer system instead of using a single shared replay buffer. The multi-buffer system allows for the Evolutionary Strategy to search freely in the search space of policies, without running the risk of overpopulating the replay buffer with poorly performing trajectories which limit the number of desirable policy behaviour examples thus negatively impacting the potential of the Deep Reinforcement Learning within the shared framework. The proposed algorithm is demonstrated to perform competitively with current Evolutionary Reinforcement Learning algorithms on MuJoCo control tasks, outperforming the well known state-of-the-art CEM-RL on 3 of the 4 environments tested.