Natural Gradient Interpretation of Rank-One Update in CMA-ES

TL;DR

This paper offers a new interpretation of the rank-one update in CMA-ES as a natural gradient step derived from a maximum a posteriori IGO framework, enhancing theoretical understanding and extensibility.

Contribution

It introduces MAP-IGO, extending IGO with a prior, and derives the rank-one update from this framework, providing a novel theoretical perspective.

Findings

The new rank-one update can be extended with an additional term.

Empirical results show the properties of the extended update.

The interpretation links CMA-ES components to natural gradient methods.

Abstract

The covariance matrix adaptation evolution strategy (CMA-ES) is a stochastic search algorithm using a multivariate normal distribution for continuous black-box optimization. In addition to strong empirical results, part of the CMA-ES can be described by a stochastic natural gradient method and can be derived from information geometric optimization (IGO) framework. However, there are some components of the CMA-ES, such as the rank-one update, for which the theoretical understanding is limited. While the rank-one update makes the covariance matrix to increase the likelihood of generating a solution in the direction of the evolution path, this idea has been difficult to formulate and interpret as a natural gradient method unlike the rank-$\mu$ update. In this work, we provide a new interpretation of the rank-one update in the CMA-ES from the perspective of the natural gradient with prior distribution. First, we propose maximum a posteriori IGO (MAP-IGO), which is the IGO framework extended to incorporate a prior distribution. Then, we derive the rank-one update from the MAP-IGO by setting the prior distribution based on the idea that the promising mean vector should exist in the direction of the evolution path. Moreover, the newly derived rank-one update is extensible, where an additional term appears in the update for the mean vector. We empirically investigate the properties of the additional term using various benchmark functions.