Quality with Just Enough Diversity in Evolutionary Policy Search

TL;DR

This paper introduces JEDi, a framework that leverages behavior information to efficiently focus evolutionary policy search, outperforming existing methods on complex exploration and control tasks.

Contribution

JEDi is a novel approach that learns the relationship between behavior and fitness to guide evaluations, improving efficiency and performance in evolutionary policy search.

Findings

JEDi outperforms QD and ES on maze and control tasks.

JEDi efficiently identifies promising search areas.

JEDi improves exploration and policy quality.

Abstract

Evolution Strategies (ES) are effective gradient-free optimization methods that can be competitive with gradient-based approaches for policy search. ES only rely on the total episodic scores of solutions in their population, from which they estimate fitness gradients for their update with no access to true gradient information. However this makes them sensitive to deceptive fitness landscapes, and they tend to only explore one way to solve a problem. Quality-Diversity methods such as MAP-Elites introduced additional information with behavior descriptors (BD) to return a population of diverse solutions, which helps exploration but leads to a large part of the evaluation budget not being focused on finding the best performing solution. Here we show that behavior information can also be leveraged to find the best policy by identifying promising search areas which can then be efficiently explored with ES. We introduce the framework of Quality with Just Enough Diversity (JEDi) which learns the relationship between behavior and fitness to focus evaluations on solutions that matter. When trying to reach higher fitness values, JEDi outperforms both QD and ES methods on hard exploration tasks like mazes and on complex control problems with large policies.