Accelerated Zero-Order SGD Method for Solving the Black Box Optimization Problem under "Overparametrization" Condition

TL;DR

This paper introduces a novel accelerated zero-order stochastic gradient descent method tailored for convex optimization problems in overparameterized settings where only function values are accessible, extending convergence guarantees to noisy scenarios.

Contribution

It proposes a gradient-free accelerated SGD algorithm using $l_2$ randomization, generalizing existing methods to handle biased and noisy function evaluations in overparameterized convex problems.

Findings

The algorithm converges under certain noise levels.

Theoretical bounds on adversarial noise are established.

Empirical verification confirms convergence results.

Abstract

This paper is devoted to solving a convex stochastic optimization problem in a overparameterization setup for the case where the original gradient computation is not available, but an objective function value can be computed. For this class of problems we provide a novel gradient-free algorithm, whose creation approach is based on applying a gradient approximation with $l_2$ randomization instead of a gradient oracle in the biased Accelerated SGD algorithm, which generalizes the convergence results of the AC-SA algorithm to the case where the gradient oracle returns a noisy (inexact) objective function value. We also perform a detailed analysis to find the maximum admissible level of adversarial noise at which we can guarantee to achieve the desired accuracy. We verify the theoretical results of convergence using a model example.