Towards a Principled Learning Rate Adaptation for Natural Evolution Strategies

TL;DR

This paper introduces a new adaptive learning rate mechanism for Natural Evolution Strategies that adjusts based on the estimated accuracy of the natural gradient, improving optimization speed and stability across different problem complexities.

Contribution

The paper proposes a novel learning rate adaptation method for NES that dynamically adjusts based on gradient estimation accuracy, enhancing performance over fixed learning rates.

Findings

The adaptive mechanism speeds up optimization on easy problems.

It provides stable and robust search on difficult, multimodal functions.

Experimental results outperform fixed learning rate approaches.

Abstract

Natural Evolution Strategies (NES) is a promising framework for black-box continuous optimization problems. NES optimizes the parameters of a probability distribution based on the estimated natural gradient, and one of the key parameters affecting the performance is the learning rate. We argue that from the viewpoint of the natural gradient method, the learning rate should be determined according to the estimation accuracy of the natural gradient. To do so, we propose a new learning rate adaptation mechanism for NES. The proposed mechanism makes it possible to set a high learning rate for problems that are relatively easy to optimize, which results in speeding up the search. On the other hand, in problems that are difficult to optimize (e.g., multimodal functions), the proposed mechanism makes it possible to set a conservative learning rate when the estimation accuracy of the natural gradient seems to be low, which results in the robust and stable search. The experimental evaluations on unimodal and multimodal functions demonstrate that the proposed mechanism works properly depending on a search situation and is effective over the existing method, i.e., using the fixed learning rate.