A Parameter-Free and Near-Optimal Zeroth-Order Algorithm for Stochastic Convex Optimization

TL;DR

This paper introduces POEM, a parameter-free zeroth-order algorithm for stochastic convex optimization, which adaptively adjusts parameters to achieve near-optimal oracle complexity and outperforms existing methods in experiments.

Contribution

The paper presents a novel parameter-free stochastic zeroth-order method with an adaptive step-size scheme, achieving near-optimal complexity for convex minimization.

Findings

POEM achieves near-optimal stochastic zeroth-order oracle complexity.

Numerical experiments show POEM outperforms existing zeroth-order methods.

The method adaptively adjusts smoothing parameters without prior knowledge.

Abstract

This paper considers zeroth-order optimization for stochastic convex minimization problem. We propose a parameter-free stochastic zeroth-order method (POEM) by introducing a step-size scheme based on the distance over finite difference and an adaptive smoothing parameter. We provide the theoretical analysis to show that POEM achieves the near-optimal stochastic zeroth-order oracle complexity. We further conduct the numerical experiments to demonstrate POEM outperforms existing zeroth-order methods in practice.