High-Probability Guarantees for Random Zeroth-Order (Stochastic) Gradient Descent

TL;DR

This paper establishes high-probability convergence guarantees for random zeroth-order gradient descent in both deterministic and stochastic settings, providing confidence bounds and query complexity analysis.

Contribution

It offers the first high-probability guarantees for zeroth-order methods, extending classical expectation-based results to probabilistic settings.

Findings

Deterministic case: finds an ε-suboptimal solution with high probability using O(dL/μ log(1/ε) + log(1/δ)) queries.

Stochastic case: achieves ε-suboptimality with high probability using O(d log(1/ε) (log(1/ε)+log(1/δ))/ε) queries.

Provides high-confidence bounds that only add a logarithmic term compared to expectation-based guarantees.

Abstract

Zeroth-order optimization aims to minimize an objective function using only function evaluations, and is therefore fundamental in black-box optimization, hyperparameter tuning, bandit learning, and adversarial machine learning. While classical zeroth-order methods are well understood in expectation, much less is known about their high-probability behavior, especially for smooth and strongly convex objectives. In this paper, we establish high-probability convergence guarantees for random zeroth-order gradient descent in both deterministic and stochastic settings. For deterministic $L$-smooth and $\mu$-strongly convex objectives of $d$-dimension, we show that the classical two-query random zeroth-order method finds an $\varepsilon$-suboptimal solution with probability at least $1-\delta$ using \[ \mathcal{O}\left( \frac{dL}{\mu}\log\frac{1}{\varepsilon} + \log\frac{1}{\delta} \right) \] function queries. Thus, compared with the standard in-expectation complexity, only an additive logarithmic dependence on the confidence parameter is needed. For stochastic objectives, under a bounded-noise condition and without assuming uniformly bounded stochastic gradients, we prove that random zeroth-order stochastic gradient descent achieves an $\varepsilon$-suboptimal solution with probability at least $1-\delta$ using \[ \mathcal{O}\left( \frac{ d\log(1/\varepsilon) \left(\log(1/\varepsilon)+\log(1/\delta)\right) }{\varepsilon} \right) \] queries. Our results provide high-confidence counterparts to classical expectation-based zeroth-order convergence guarantees and clarify the additional cost required to obtain reliable performance guarantees.