Regret-Aware Black-Box Optimization with Natural Gradients, Trust-Regions and Entropy Control

TL;DR

This paper introduces an improved regret-aware black-box optimization method that combines natural gradients, trust regions, and entropy control, outperforming ranking-based methods especially in reinforcement learning tasks.

Contribution

The authors enhance the Model-based Relative Entropy Stochastic Search (MORE) algorithm by decoupling mean and covariance updates, introducing an entropy scheduling technique, and simplifying model learning, leading to faster convergence and better regret performance.

Findings

Outperforms ranking-based methods in regret on RL tasks.

Achieves competitive results on standard benchmark functions.

Faster convergence due to improved entropy scheduling.

Abstract

Most successful stochastic black-box optimizers, such as CMA-ES, use rankings of the individual samples to obtain a new search distribution. Yet, the use of rankings also introduces several issues such as the underlying optimization objective is often unclear, i.e., we do not optimize the expected fitness. Further, while these algorithms typically produce a high-quality mean estimate of the search distribution, the produced samples can have poor quality as these algorithms are ignorant of the regret. Lastly, noisy fitness function evaluations may result in solutions that are highly sub-optimal on expectation. In contrast, stochastic optimizers that are motivated by policy gradients, such as the Model-based Relative Entropy Stochastic Search (MORE) algorithm, directly optimize the expected fitness function without the use of rankings. MORE can be derived by applying natural policy gradients and compatible function approximation, and is using information theoretic constraints to ensure the stability of the policy update. While MORE does not suffer from the listed limitations, it often cannot achieve state of the art performance in comparison to ranking based methods. We improve MORE by decoupling the update of the mean and covariance of the search distribution allowing for more aggressive updates on the mean while keeping the update on the covariance conservative, an improved entropy scheduling technique based on an evolution path which results in faster convergence and a simplified and more effective model learning approach in comparison to the original paper. We compare our algorithm to state of the art black-box optimization algorithms on standard optimization tasks as well as on episodic RL tasks in robotics where it is also crucial to have small regret. We obtain competitive results on benchmark functions and clearly outperform ranking-based methods in terms of regret on the RL tasks.