Efficacy of Modern Neuro-Evolutionary Strategies for Continuous Control Optimization

TL;DR

This paper evaluates modern neuro-evolutionary strategies for continuous control, showing their effectiveness, scalability, and robustness, while highlighting differences in reward function optimization between reinforcement learning and evolutionary methods.

Contribution

It provides a comprehensive comparison of neuro-evolutionary algorithms, demonstrating the superior or equal performance of OpenAI-ES and revealing biases in reward function optimization.

Findings

Neuro-evolutionary methods are effective and scalable.

OpenAI-ES outperforms or matches other algorithms.

Reward functions differ in effectiveness between RL and evolutionary strategies.

Abstract

We analyze the efficacy of modern neuro-evolutionary strategies for continuous control optimization. Overall, the results collected on a wide variety of qualitatively different benchmark problems indicate that these methods are generally effective and scale well with respect to the number of parameters and the complexity of the problem. Moreover, they are relatively robust with respect to the setting of hyper-parameters. The comparison of the most promising methods indicates that the OpenAI-ES algorithm outperforms or equals the other algorithms on all considered problems. Moreover, we demonstrate how the reward functions optimized for reinforcement learning methods are not necessarily effective for evolutionary strategies and vice versa. This finding can lead to reconsideration of the relative efficacy of the two classes of algorithm since it implies that the comparisons performed to date are biased toward one or the other class.