Back to Basics: Benchmarking Canonical Evolution Strategies for Playing Atari

TL;DR

This paper shows that a simple canonical Evolution Strategies algorithm can match or outperform specialized methods in Atari game benchmarks, highlighting the potential for further improvements by integrating ES and RL techniques.

Contribution

Demonstrates that basic canonical ES can achieve competitive performance on Atari benchmarks, challenging the notion that complex ES variants are necessary.

Findings

Basic ES matches or exceeds specialized ES performance.

ES algorithms exploit environment differently than RL.

Potential for combining ES and RL for improved results.

Abstract

Evolution Strategies (ES) have recently been demonstrated to be a viable alternative to reinforcement learning (RL) algorithms on a set of challenging deep RL problems, including Atari games and MuJoCo humanoid locomotion benchmarks. While the ES algorithms in that work belonged to the specialized class of natural evolution strategies (which resemble approximate gradient RL algorithms, such as REINFORCE), we demonstrate that even a very basic canonical ES algorithm can achieve the same or even better performance. This success of a basic ES algorithm suggests that the state-of-the-art can be advanced further by integrating the many advances made in the field of ES in the last decades. We also demonstrate qualitatively that ES algorithms have very different performance characteristics than traditional RL algorithms: on some games, they learn to exploit the environment and perform much better while on others they can get stuck in suboptimal local minima. Combining their strengths with those of traditional RL algorithms is therefore likely to lead to new advances in the state of the art.