An Algorithm with Optimal Dimension-Dependence for Zero-Order Nonsmooth Nonconvex Stochastic Optimization

TL;DR

This paper introduces a new zero-order stochastic optimization algorithm for nonsmooth, nonconvex functions that achieves the optimal dimension dependence and convergence rate, improving upon previous methods.

Contribution

It provides the first algorithm with optimal dimension dependence for zero-order nonsmooth nonconvex stochastic optimization, resolving an open problem.

Findings

Achieves complexity $O(d delta^{-1} epsilon^{-3})$, optimal in dimension and accuracy.

Proves nonsmooth optimization is as easy as smooth in the stochastic zero-order setting.

Provides algorithms with convergence guarantees in expectation and high probability.

Abstract

We study the complexity of producing $(\delta,\epsilon)$-stationary points of Lipschitz objectives which are possibly neither smooth nor convex, using only noisy function evaluations. Recent works proposed several stochastic zero-order algorithms that solve this task, all of which suffer from a dimension-dependence of $\Omega(d^{3/2})$ where $d$ is the dimension of the problem, which was conjectured to be optimal. We refute this conjecture by providing a faster algorithm that has complexity $O(d\delta^{-1}\epsilon^{-3})$, which is optimal (up to numerical constants) with respect to $d$ and also optimal with respect to the accuracy parameters $\delta,\epsilon$, thus solving an open question due to Lin et al. (NeurIPS'22). Moreover, the convergence rate achieved by our algorithm is also optimal for smooth objectives, proving that in the nonconvex stochastic zero-order setting, nonsmooth optimization is as easy as smooth optimization. We provide algorithms that achieve the aforementioned convergence rate in expectation as well as with high probability. Our analysis is based on a simple yet powerful lemma regarding the Goldstein-subdifferential set, which allows utilizing recent advancements in first-order nonsmooth nonconvex optimization.