Refining Adaptive Zeroth-Order Optimization at Ease

TL;DR

This paper introduces R-AdaZO, a novel adaptive zeroth-order optimization method that leverages variance reduction techniques to improve convergence speed and stability in black-box and resource-constrained scenarios.

Contribution

It provides the first variance reduction analysis for first moment estimates in ZO optimization and develops a variance-aware convergence framework for adaptive ZO methods.

Findings

R-AdaZO achieves faster convergence than ZO-AdaMM.

Theoretical analysis confirms variance reduction benefits.

Experiments demonstrate improved performance in black-box attacks and LLM fine-tuning.

Abstract

Recently, zeroth-order (ZO) optimization plays an essential role in scenarios where gradient information is inaccessible or unaffordable, such as black-box systems and resource-constrained environments. While existing adaptive methods such as ZO-AdaMM have shown promise, they are fundamentally limited by their underutilization of moment information during optimization, usually resulting in underperforming convergence. To overcome these limitations, this paper introduces Refined Adaptive Zeroth-Order Optimization (R-AdaZO). Specifically, we first show the untapped variance reduction effect of first moment estimate on ZO gradient estimation, which improves the accuracy and stability of ZO updates. We then refine the second moment estimate based on these variance-reduced gradient estimates to better capture the geometry of the optimization landscape, enabling a more effective scaling of ZO updates. We present rigorous theoretical analysis to show (a) the first analysis to the variance reduction of first moment estimate in ZO optimization, (b) the improved second moment estimates with a more accurate approximation of its variance-free ideal, (c) the first variance-aware convergence framework for adaptive ZO methods, which may be of independent interest, and (d) the faster convergence of R-AdaZO than existing baselines like ZO-AdaMM. Our extensive experiments, including synthetic problems, black-box adversarial attack, and memory-efficient fine-tuning of large language models (LLMs), further verify the superior convergence of R-AdaZO, indicating that R-AdaZO offers an improved solution for real-world ZO optimization challenges.