Improved Iteration Complexity in Black-Box Optimization Problems under Higher Order Smoothness Function Condition

TL;DR

This paper introduces a new zero-order optimization method for strongly convex functions with higher order smoothness, achieving optimal iteration complexity in stochastic black-box settings.

Contribution

It proposes a novel accelerated batched stochastic gradient descent method that attains optimal iteration complexity for higher order smoothness functions in black-box optimization.

Findings

Achieves optimal iteration complexity bounds.

Analyzes maximum noise level considering batch size and smoothness order.

Provides theoretical guarantees for the proposed method.

Abstract

This paper is devoted to the study (common in many applications) of the black-box optimization problem, where the black-box represents a gradient-free oracle $\tilde{f} = f(x) + \xi$ providing the objective function value with some stochastic noise. Assuming that the objective function is $\mu$-strongly convex, and also not just $L$-smooth, but has a higher order of smoothness ($\beta \geq 2$) we provide a novel optimization method: Zero-Order Accelerated Batched Stochastic Gradient Descent, whose theoretical analysis closes the question regarding the iteration complexity, achieving optimal estimates. Moreover, we provide a thorough analysis of the maximum noise level, and show under which condition the maximum noise level will take into account information about batch size $B$ as well as information about the smoothness order of the function $\beta$.