Stochastic Zeroth-Order Optimization Under Heavy-Tailed Noise

TL;DR

This paper introduces RSC-ZO, a robust zeroth-order optimization method that achieves high-probability guarantees under heavy-tailed noise, matching first-order rates and extending classical bounds.

Contribution

The paper proposes RSC-ZO, a novel scalar-clipped zeroth-order method that handles heavy-tailed noise with high-probability guarantees, a significant advancement over classical ZO theory.

Findings

RSC-ZO finds an ε-stationary point with high probability using rac{d^{p/(2(p-1))}}{ ext{evaluations}}.

At p=2, the method achieves rac{d \u2212 4}{ ext{evaluations}}, matching classical bounds.

The analysis includes a momentum variant and explores batch-size and stepsize tradeoffs.

Abstract

We study stochastic zeroth-order (ZO) optimization of smooth nonconvex objectives under heavy-tailed sample-gradient noise. This regime is motivated by empirical evidence that gradient noise in modern machine learning can violate the bounded-variance assumptions used in classical ZO theory. While first-order methods have optimal rates under bounded $p$-th moment noise for $p\in(1,2]$, analogous high-probability guarantees for nonconvex ZO methods are much less understood. The ZO setting is not a direct corollary of first-order theory. First-order methods observe stochastic gradients, whereas derivative-free methods only query noisy function values and build finite-difference estimates. Thus, weak-$L_p$ control of $\nabla F(x;\xi)-\nabla f(x)$ must first be transferred to scalar directional estimates. We propose the Robust Scalar-Clipped Zeroth-Order method (RSC-ZO), a two-point method that clips each scalar directional derivative before aggregation. Under sample-wise smoothness and a weak-$L_p$ tail condition on the sample-gradient noise, RSC-ZO finds an $\varepsilon$-stationary point with high probability using $$ \widetilde{O}\!\left( d^{\frac{p}{2(p-1)}}\varepsilon^{-\frac{3p-2}{p-1}} \right) $$ noisy function evaluations. This matches the optimal first-order $\varepsilon$-dependence. At $p=2$, the bound becomes $\widetilde{O}(d\varepsilon^{-4})$, matching the classical stochastic ZO dimension--accuracy dependence, but with a high-probability guarantee and under a weaker weak-$L_2$ condition that can allow infinite variance. We also analyze a momentum variant and quantify its batch-size/stepsize tradeoff.