Why Does Adaptive Zeroth-Order Optimization Work?

TL;DR

This paper provides a theoretical explanation for the effectiveness of adaptive zeroth-order optimization methods, showing they are closely related to gradient norms and offering convergence guarantees under generalized smoothness conditions.

Contribution

It introduces a theoretical analysis linking empirical standard deviation to gradient norms and establishes convergence rates for adaptive ZO methods under generalized smoothness.

Findings

Empirical standard deviation closely approximates gradient norm with high probability.

Adaptive ZO methods achieve faster convergence than fixed-step methods.

Explicit query complexity bounds are derived for both deterministic and stochastic cases.

Abstract

Zeroth-order (ZO) optimization is popular in real-world applications that accessing the gradient information is expensive or unavailable. Recently, adaptive ZO methods that normalize gradient estimators by the empirical standard deviation of function values have achieved strong practical performance, particularly in fine-tuning the large language model. However, the theoretical understanding of such strategy remains limited. In this work, we show that the empirical standard deviation is, with high probability, closely proportional to the norm of the (stochastic) gradient. Based on this insight, we analyze adaptive ZO methods under the generalized $(L_0,L_1)$-smoothness condition with respect to the matrix norm. We establish explicit convergence rates and query complexity bounds for both deterministic and stochastic settings, demonstrating that adaptive ZO methods achieve the faster convergence and the improved query efficiency compared to the vanilla ZO methods with fixed-step.