Accelerating Single-Point Zeroth-Order Optimization with Regression-Based Gradient Surrogates

TL;DR

This paper introduces ReSZO, a regression-based single-point zeroth-order optimization method that constructs surrogate functions to significantly improve convergence speed in black-box optimization tasks.

Contribution

ReSZO is a novel framework that uses regression to create surrogate functions, enhancing convergence rates of single-point ZO methods with theoretical analysis and practical improvements.

Findings

ReSZO converges two to three times faster than two-point ZO in experiments.

Theoretical convergence rates of ReSZO are comparable to two-point ZO methods.

ReSZO effectively reduces gradient estimation variance in black-box optimization.

Abstract

Zeroth-order optimization (ZO) is widely used for solving black-box optimization and control problems. In particular, single-point ZO (SZO) is well-suited to online or dynamic problem settings due to its requirement of only a single function evaluation per iteration. However, SZO suffers from high gradient estimation variance and slow convergence, which severely limit its practical applicability. To overcome this limitation, we propose a novel yet simple SZO framework termed regression-based SZO (ReSZO), which substantially enhances the convergence rate. Specifically, ReSZO constructs a surrogate function via regression using historical function evaluations and employs the gradient of this surrogate function for iterative updates. Two instantiations of ReSZO, which fit linear and quadratic surrogate functions respectively, are introduced. Moreover, we provide a non-asymptotic convergence analysis for the linear instantiation of ReSZO, showing that its convergence rates are comparable to those of two-point ZO methods. Extensive numerical experiments demonstrate that ReSZO empirically converges two to three times faster than two-point ZO in terms of function query complexity.