Evolutionary Reinforcement Learning via Cooperative Coevolutionary Negatively Correlated Search

TL;DR

This paper introduces a cooperative coevolution framework for Negatively Correlated Search to efficiently optimize large-scale neural policies in reinforcement learning, demonstrating superior performance on Atari games.

Contribution

It proposes a scalable NCS-friendly cooperative coevolution approach that enhances exploration in high-dimensional RL policy optimization tasks.

Findings

Outperforms three state-of-the-art deep RL methods

Reduces computational time by 50%

Successfully explores a 1.7 million-dimensional search space

Abstract

Evolutionary algorithms (EAs) have been successfully applied to optimize the policies for Reinforcement Learning (RL) tasks due to their exploration ability. The recently proposed Negatively Correlated Search (NCS) provides a distinct parallel exploration search behavior and is expected to facilitate RL more effectively. Considering that the commonly adopted neural policies usually involves millions of parameters to be optimized, the direct application of NCS to RL may face a great challenge of the large-scale search space. To address this issue, this paper presents an NCS-friendly Cooperative Coevolution (CC) framework to scale-up NCS while largely preserving its parallel exploration search behavior. The issue of traditional CC that can deteriorate NCS is also discussed. Empirical studies on 10 popular Atari games show that the proposed method can significantly outperform three state-of-the-art deep RL methods with 50% less computational time by effectively exploring a 1.7 million-dimensional search space.