Fully Adaptive Zeroth-Order Method for Minimizing Functions with Compressible Gradients

TL;DR

This paper introduces an adaptive zeroth-order optimization method that efficiently minimizes functions with compressible gradients, achieving improved complexity bounds without prior knowledge of gradient sparsity or Lipschitz constants.

Contribution

The paper presents a novel adaptive zeroth-order method that exploits gradient compressibility, providing improved theoretical complexity bounds and practical efficiency without requiring prior parameter knowledge.

Findings

Achieves $O(n^{2} ext{epsilon}^{-2})$ function evaluations for stationary points.

Under gradient compressibility, reduces complexity to $O(s ext{log}(n) ext{epsilon}^{-2})$ evaluations.

Numerical results show significant performance improvements over non-adaptive methods.

Abstract

We propose an adaptive zeroth-order method for minimizing differentiable functions with $L$-Lipschitz continuous gradients. The method is designed to take advantage of the eventual compressibility of the gradient of the objective function, but it does not require knowledge of the approximate sparsity level $s$ or the Lipschitz constant $L$ of the gradient. We show that the new method performs no more than $O\left(n^{2}\epsilon^{-2}\right)$ function evaluations to find an $\epsilon$-approximate stationary point of an objective function with $n$ variables. Assuming additionally that the gradients of the objective function are compressible, we obtain an improved complexity bound of $O\left(s\log\left(n\right)\epsilon^{-2}\right)$ function evaluations, which holds with high probability. Preliminary numerical results illustrate the efficiency of the proposed method and demonstrate that it can significantly outperform its non-adaptive counterpart.