Stochastic Zeroth-order Optimization in High Dimensions

TL;DR

This paper introduces two algorithms for high-dimensional convex optimization using stochastic zeroth-order queries, achieving convergence rates that depend only logarithmically on the dimension, and demonstrates their superior performance over classical methods.

Contribution

The paper proposes novel algorithms leveraging sparsity assumptions for efficient high-dimensional zeroth-order optimization, with theoretical convergence guarantees.

Findings

Algorithms outperform classical zeroth-order methods in high dimensions.

Convergence rates depend logarithmically on ambient dimension.

Empirical results validate theoretical advantages.

Abstract

We consider the problem of optimizing a high-dimensional convex function using stochastic zeroth-order queries. Under sparsity assumptions on the gradients or function values, we present two algorithms: a successive component/feature selection algorithm and a noisy mirror descent algorithm using Lasso gradient estimates, and show that both algorithms have convergence rates that de- pend only logarithmically on the ambient dimension of the problem. Empirical results confirm our theoretical findings and show that the algorithms we design outperform classical zeroth-order optimization methods in the high-dimensional setting.