Highly Smoothness Zero-Order Methods for Solving Optimization Problems under PL Condition

TL;DR

This paper introduces advanced zero-order optimization methods leveraging higher smoothness and kernel-based approximations to improve convergence under the Polyak--Lojasiewicz condition, even with noisy function evaluations.

Contribution

It develops novel zero-order algorithms that outperform existing methods by exploiting higher smoothness and handling adversarial noise under the PL condition.

Findings

Improved convergence rates for zero-order methods under PL condition.

Effective handling of noisy function evaluations in optimization.

Validation through solving nonlinear systems.

Abstract

In this paper, we study the black box optimization problem under the Polyak--Lojasiewicz (PL) condition, assuming that the objective function is not just smooth, but has higher smoothness. By using "kernel-based" approximation instead of the exact gradient in Stochastic Gradient Descent method, we improve the best known results of convergence in the class of gradient-free algorithms solving problem under PL condition. We generalize our results to the case where a zero-order oracle returns a function value at a point with some adversarial noise. We verify our theoretical results on the example of solving a system of nonlinear equations.