Curiosity creates Diversity in Policy Search

TL;DR

This paper introduces Curiosity-ES, an evolutionary strategy that uses intrinsic motivation to promote diversity in policy search, especially in reward-sparse environments, leading to multiple reward-yielding policies.

Contribution

It proposes a novel evolutionary strategy that incorporates Curiosity as a fitness metric, outperforming traditional diversity metrics like Novelty.

Findings

Curiosity-ES generates higher diversity without explicit diversity criteria.

Curiosity-ES finds multiple reward-yielding policies.

Curiosity outperforms Novelty in promoting diverse behaviors.

Abstract

When searching for policies, reward-sparse environments often lack sufficient information about which behaviors to improve upon or avoid. In such environments, the policy search process is bound to blindly search for reward-yielding transitions and no early reward can bias this search in one direction or another. A way to overcome this is to use intrinsic motivation in order to explore new transitions until a reward is found. In this work, we use a recently proposed definition of intrinsic motivation, Curiosity, in an evolutionary policy search method. We propose Curiosity-ES, an evolutionary strategy adapted to use Curiosity as a fitness metric. We compare Curiosity with Novelty, a commonly used diversity metric, and find that Curiosity can generate higher diversity over full episodes without the need for an explicit diversity criterion and lead to multiple policies which find reward.