A Zeroth-order Proximal Stochastic Gradient Method for Weakly Convex Stochastic Optimization

TL;DR

This paper introduces a zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization, enabling optimization without gradient information and demonstrating effectiveness in hyper-parameter tuning and PDE-constrained problems.

Contribution

It proposes a novel zeroth-order proximal stochastic gradient algorithm using Gaussian smoothing for weakly convex problems, with proven convergence rates and practical applications.

Findings

Achieves state-of-the-art convergence rates for zeroth-order methods.

Effectively tunes hyper-parameters automatically in complex optimization tasks.

Demonstrates empirical success on phase retrieval and PDE-constrained problems.

Abstract

In this paper we analyze a zeroth-order proximal stochastic gradient method suitable for the minimization of weakly convex stochastic optimization problems. We consider nonsmooth and nonlinear stochastic composite problems, for which (sub-)gradient information might be unavailable. The proposed algorithm utilizes the well-known Gaussian smoothing technique, which yields unbiased zeroth-order gradient estimators of a related partially smooth surrogate problem (in which one of the two nonsmooth terms in the original problem's objective is replaced by a smooth approximation). This allows us to employ a standard proximal stochastic gradient scheme for the approximate solution of the surrogate problem, which is determined by a single smoothing parameter, and without the utilization of first-order information. We provide state-of-the-art convergence rates for the proposed zeroth-order method using minimal assumptions. The proposed scheme is numerically compared against alternative zeroth-order methods as well as a stochastic sub-gradient scheme on a standard phase retrieval problem. Further, we showcase the usefulness and effectiveness of our method for the unique setting of automated hyper-parameter tuning. In particular, we focus on automatically tuning the parameters of optimization algorithms by minimizing a novel heuristic model. The proposed approach is tested on a proximal alternating direction method of multipliers for the solution of $\mathcal{L}_1/\mathcal{L}_2$-regularized PDE-constrained optimal control problems, with evident empirical success.