Mirror Natural Evolution Strategies

TL;DR

This paper introduces MiNES, a zeroth-order optimization algorithm that estimates both gradient and Hessian information using function queries, with proven convergence rates and a novel reparameterized objective.

Contribution

It proposes a new algorithm, MiNES, which effectively approximates the Hessian inverse and demonstrates its convergence properties in zeroth-order optimization.

Findings

Covariance matrix of MiNES converges to the inverse Hessian.

Convergence rate of MiNES is (1/k) with respect to iterations.

Explicit convergence bounds are provided for the algorithm.

Abstract

The zeroth-order optimization has been widely used in machine learning applications. However, the theoretical study of the zeroth-order optimization focus on the algorithms which approximate (first-order) gradients using (zeroth-order) function value difference at a random direction. The theory of algorithms which approximate the gradient and Hessian information by zeroth-order queries is much less studied. In this paper, we focus on the theory of zeroth-order optimization which utilizes both the first-order and second-order information approximated by the zeroth-order queries. We first propose a novel reparameterized objective function with parameters $(\mu, \Sigma)$. This reparameterized objective function achieves its optimum at the minimizer and the Hessian inverse of the original objective function respectively, but with small perturbations. Accordingly, we propose a new algorithm to minimize our proposed reparameterized objective, which we call \texttt{MiNES} (mirror descent natural evolution strategy). We show that the estimated covariance matrix of \texttt{MiNES} converges to the inverse of Hessian matrix of the objective function with a convergence rate $\widetilde{\mathcal{O}}(1/k)$, where $k$ is the iteration number and $\widetilde{\mathcal{O}}(\cdot)$ hides the constant and $\log$ terms. We also provide the explicit convergence rate of \texttt{MiNES} and how the covariance matrix promotes the convergence rate.