The Black-Box Optimization Problem: Zero-Order Accelerated Stochastic Method via Kernel Approximation

TL;DR

This paper introduces a zero-order accelerated stochastic gradient method that leverages kernel approximation and higher-order smoothness to improve convergence in black-box optimization problems with noisy function evaluations.

Contribution

It proposes a novel ZO-AccSGD algorithm that exploits higher-order smoothness via kernel approximation, outperforming existing methods in convergence and noise tolerance.

Findings

Improved iteration complexity over state-of-the-art algorithms.

Theoretical bounds on maximum noise level for achieving desired accuracy.

Validation on machine learning functions confirms practical effectiveness.

Abstract

In this paper, we study the standard formulation of an optimization problem when the computation of gradient is not available. Such a problem can be classified as a "black box" optimization problem, since the oracle returns only the value of the objective function at the requested point, possibly with some stochastic noise. Assuming convex, and higher-order of smoothness of the objective function, this paper provides a zero-order accelerated stochastic gradient descent (ZO-AccSGD) method for solving this problem, which exploits the higher-order of smoothness information via kernel approximation. As theoretical results, we show that the ZO-AccSGD algorithm proposed in this paper improves the convergence results of state-of-the-art (SOTA) algorithms, namely the estimate of iteration complexity. In addition, our theoretical analysis provides an estimate of the maximum allowable noise level at which the desired accuracy can be achieved. Validation of our theoretical results is demonstrated both on the model function and on functions of interest in the field of machine learning. We also provide a discussion in which we explain the results obtained and the superiority of the proposed algorithm over SOTA algorithms for solving the original problem.