Imaginary Zeroth-Order Optimization

TL;DR

This paper explores an innovative approach to zeroth-order optimization by leveraging complex domain techniques to mitigate numerical cancellation issues, providing theoretical analysis and practical results in noisy, strongly-convex, and non-convex settings.

Contribution

It introduces a novel complex domain method for zeroth-order optimization, addressing numerical cancellation and analyzing its effectiveness under computational noise.

Findings

Effective in strongly-convex optimization with noise

Provides non-asymptotic convergence results

Demonstrates practical benefits through numerical experiments

Abstract

Zeroth-order optimization methods are developed to overcome the practical hurdle of having knowledge of explicit derivatives. Instead, these schemes work with merely access to noisy functions evaluations. One of the predominant approaches is to mimic first-order methods by means of some gradient estimator. The theoretical limitations are well-understood, yet, as most of these methods rely on finite-differencing for shrinking differences, numerical cancellation can be catastrophic. The numerical community developed an efficient method to overcome this by passing to the complex domain. This approach has been recently adopted by the optimization community and in this work we analyze the practically relevant setting of dealing with computational noise. To exemplify the possibilities we focus on the strongly-convex optimization setting and provide a variety of non-asymptotic results, corroborated by numerical experiments, and end with local non-convex optimization.