Evolution Strategies for Deep RL pretraining

TL;DR

This paper evaluates the effectiveness of evolution strategies as a pretraining method for deep reinforcement learning across various tasks, comparing their performance to traditional DRL methods.

Contribution

It provides a systematic analysis of ES as a pretraining step, highlighting its limited benefits for complex environments compared to DRL.

Findings

ES do not train faster than DRL in most cases

Pretraining with ES benefits only simple tasks like Flappy Bird

ES shows minimal or no improvement in complex environments like Breakout and MuJoCo

Abstract

Although Deep Reinforcement Learning has proven highly effective for complex decision-making problems, it demands significant computational resources and careful parameter adjustment in order to develop successful strategies. Evolution strategies offer a more straightforward, derivative-free approach that is less computationally costly and simpler to deploy. However, ES generally do not match the performance levels achieved by DRL, which calls into question their suitability for more demanding scenarios. This study examines the performance of ES and DRL across tasks of varying difficulty, including Flappy Bird, Breakout and Mujoco environments, as well as whether ES could be used for initial training to enhance DRL algorithms. The results indicate that ES do not consistently train faster than DRL. When used as a preliminary training step, they only provide benefits in less complex environments (Flappy Bird) and show minimal or no improvement in training efficiency or stability across different parameter settings when applied to more sophisticated tasks (Breakout and MuJoCo Walker).