Highly-Smooth Zero-th Order Online Optimization Vianney Perchet

TL;DR

This paper explores how high degrees of smoothness in convex functions can be leveraged in zero-th order online optimization to improve estimation rates, matching gradient-based methods under certain conditions.

Contribution

It demonstrates that high-order smoothness can be exploited to enhance zero-th order optimization, achieving bounds comparable to gradient-based algorithms for infinitely differentiable functions.

Findings

High-order smoothness improves estimation rates in zero-th order optimization.

For infinitely differentiable functions, bounds match gradient-based algorithms with an extra dimension factor.

Results apply to both convex and strongly-convex functions in online settings.

Abstract

The minimization of convex functions which are only available through partial and noisy information is a key methodological problem in many disciplines. In this paper we consider convex optimization with noisy zero-th order information, that is noisy function evaluations at any desired point. We focus on problems with high degrees of smoothness, such as logistic regression. We show that as opposed to gradient-based algorithms, high-order smoothness may be used to improve estimation rates, with a precise dependence of our upper-bounds on the degree of smoothness. In particular, we show that for infinitely differentiable functions, we recover the same dependence on sample size as gradient-based algorithms, with an extra dimension-dependent factor. This is done for both convex and strongly-convex functions, with finite horizon and anytime algorithms. Finally, we also recover similar results in the online optimization setting.