Subspace-based Approximate Hessian Method for Zeroth-Order Optimization

TL;DR

This paper introduces ZO-SAH, a zeroth-order optimization method that efficiently approximates Hessians in low-dimensional subspaces, leading to faster convergence in high-dimensional problems.

Contribution

The paper proposes a novel subspace-based Hessian approximation technique for zeroth-order optimization, reducing function evaluations and accelerating convergence.

Findings

ZO-SAH outperforms existing zeroth-order methods in convergence speed.

The subspace-switching strategy effectively reuses function evaluations.

Experiments on benchmarks show significant improvements in optimization efficiency.

Abstract

Zeroth-order optimization addresses problems where gradient information is inaccessible or impractical to compute. While most existing methods rely on first-order approximations, incorporating second-order (curvature) information can, in principle, significantly accelerate convergence. However, the high cost of function evaluations required to estimate Hessian matrices often limits practical applicability. We present the subspace-based approximate Hessian (ZO-SAH) method, a zeroth-order optimization algorithm that mitigates these costs by focusing on randomly selected two-dimensional subspaces. Within each subspace, ZO-SAH estimates the Hessian by fitting a quadratic polynomial to the objective function and extracting its second-order coefficients. To further reduce function-query costs, ZO-SAH employs a periodic subspace-switching strategy that reuses function evaluations across optimization steps. Experiments on eight benchmark datasets, including logistic regression and deep neural network training tasks, demonstrate that ZO-SAH achieves significantly faster convergence than existing zeroth-order methods.