On low complexity Acceleration Techniques for Randomized Optimization: Supplementary Online Material

TL;DR

This paper empirically evaluates a simple accelerated random search method (SARP) that uses low complexity techniques inspired by convex optimization, demonstrating its effectiveness without gradient information and exploring its relation to evolution path strategies.

Contribution

The paper implements and tests SARP with adaptive step size control, and introduces CMA-EP, a low-memory scheme based on evolution paths, analyzing their performance and theoretical connections.

Findings

SARP effectively accelerates randomized optimization without gradients.

CMA-EP's performance depends heavily on the objective function's spectra.

SARP outperforms CMA-EP in various scenarios.

Abstract

Recently it was shown by Nesterov (2011) that techniques form convex optimization can be used to successfully accelerate simple derivative-free randomized optimization methods. The appeal of those schemes lies in their low complexity, which is only $\Theta(n)$ per iteration---compared to $\Theta(n^2)$ for algorithms storing second-order information or covariance matrices. From a high-level point of view, those accelerated schemes employ correlations between successive iterates---a concept looking similar to the evolution path used in Covariance Matrix Adaptation Evolution Strategies (CMA-ES). In this contribution, we (i) implement and empirically test a simple accelerated random search scheme (SARP). Our study is the first to provide numerical evidence that SARP can effectively be implemented with adaptive step size control and does not require access to gradient or advanced line search oracles. We (ii) try to empirically verify the supposed analogy between the evolution path and SARP. We propose an algorithm CMA-EP that uses only the evolution path to bias the search. This algorithm can be generalized to a family of low memory schemes, with complexity $\Theta(mn)$ per iteration, following a recent approach by Loshchilov (2014). The study shows that the performance of CMA-EP heavily depends on the spectra of the objective function and thus it cannot accelerate as consistently as SARP.