Adaptive Zeroth-Order Optimisation of Nonconvex Composite Objectives

TL;DR

This paper introduces an adaptive zeroth-order optimization algorithm for non-convex composite objectives that leverages low-dimensional structure and non-Euclidean geometry to improve convergence and reduce complexity.

Contribution

It presents a novel adaptive stochastic mirror descent method using Rademacher sampling, enhancing zeroth-order optimization for non-convex problems with low-dimensional structure.

Findings

Reduced complexity dependence on dimensionality

Adaptive stepsizes eliminate need for hyperparameter tuning

Effective gradient estimation with Rademacher sampling

Abstract

In this paper, we propose and analyze algorithms for zeroth-order optimization of non-convex composite objectives, focusing on reducing the complexity dependence on dimensionality. This is achieved by exploiting the low dimensional structure of the decision set using the stochastic mirror descent method with an entropy alike function, which performs gradient descent in the space equipped with the maximum norm. To improve the gradient estimation, we replace the classic Gaussian smoothing method with a sampling method based on the Rademacher distribution and show that the mini-batch method copes with the non-Euclidean geometry. To avoid tuning hyperparameters, we analyze the adaptive stepsizes for the general stochastic mirror descent and show that the adaptive version of the proposed algorithm converges without requiring prior knowledge about the problem.