Improving Sample Efficiency in Evolutionary RL Using Off-Policy Ranking

TL;DR

This paper introduces an off-policy ranking method for Evolution Strategies in Reinforcement Learning, significantly reducing sample requirements while maintaining performance, demonstrated on MuJoCo tasks with ARS.

Contribution

It proposes a novel off-policy ranking approach using local fitness approximation, improving sample efficiency in ES-based RL methods.

Findings

Achieves 30% reduction in data usage compared to original ARS.

Maintains similar reward thresholds with less data.

Outperforms recent Trust Region ES in experiments.

Abstract

Evolution Strategy (ES) is a powerful black-box optimization technique based on the idea of natural evolution. In each of its iterations, a key step entails ranking candidate solutions based on some fitness score. For an ES method in Reinforcement Learning (RL), this ranking step requires evaluating multiple policies. This is presently done via on-policy approaches: each policy's score is estimated by interacting several times with the environment using that policy. This leads to a lot of wasteful interactions since, once the ranking is done, only the data associated with the top-ranked policies is used for subsequent learning. To improve sample efficiency, we propose a novel off-policy alternative for ranking, based on a local approximation for the fitness function. We demonstrate our idea in the context of a state-of-the-art ES method called the Augmented Random Search (ARS). Simulations in MuJoCo tasks show that, compared to the original ARS, our off-policy variant has similar running times for reaching reward thresholds but needs only around 70% as much data. It also outperforms the recent Trust Region ES. We believe our ideas should be extendable to other ES methods as well.