Parallel Exploration via Negatively Correlated Search

TL;DR

This paper introduces a principled version of Negatively Correlated Search (NCS) that enhances exploration in reinforcement learning by explicitly maximizing diversity and solution quality, demonstrated through Atari game experiments.

Contribution

The paper provides a theoretical foundation for NCS, linking it to population diversity and solution quality maximization, and applies it to complex reinforcement learning tasks.

Findings

NCS effectively improves exploration in reinforcement learning.

Empirical results show NCS outperforms state-of-the-art methods.

NCS enhances performance on Atari game benchmarks.

Abstract

Effective exploration is a key to successful search. The recently proposed Negatively Correlated Search (NCS) tries to achieve this by parallel exploration, where a set of search processes are driven to be negatively correlated so that different promising areas of the search space can be visited simultaneously. Various applications have verified the advantages of such novel search behaviors. Nevertheless, the mathematical understandings are still lacking as the previous NCS was mostly devised by intuition. In this paper, a more principled NCS is presented, explaining that the parallel exploration is equivalent to the explicit maximization of both the population diversity and the population solution qualities, and can be optimally obtained by partially gradient descending both models with respect to each search process. For empirical assessments, the reinforcement learning tasks that largely demand exploration ability is considered. The new NCS is applied to the popular reinforcement learning problems, i.e., playing Atari games, to directly train a deep convolution network with 1.7 million connection weights in the environments with uncertain and delayed rewards. Empirical results show that the significant advantages of NCS over the compared state-of-the-art methods can be highly owed to the effective parallel exploration ability.