Inexact Zeroth-Order Nonsmooth and Nonconvex Stochastic Composite Optimization and Applications

TL;DR

This paper introduces an inexact zeroth-order method for nonsmooth, nonconvex stochastic composite optimization that converges to a stationary point under mild conditions, broadening applicability to complex stochastic problems.

Contribution

The paper proposes a novel inexact zeroth-order algorithm with convergence guarantees for nonsmooth, nonconvex stochastic composite problems, extending existing methods to more general settings.

Findings

Converges non-asymptotically to a stationary point

Applicable to large classes of nonconvex stochastic problems

Enables solving problems beyond current state-of-the-art methods

Abstract

In this paper we present an inexact zeroth-order method suitable for the solution nonsmooth and nonconvex stochastic composite optimization problems, in which the objective is split into a real-valued Lipschitz continuous stochastic function and an extended-valued (deterministic) proper, closed, and convex one. The algorithm operates under inexact oracles providing noisy (and biased) stochastic evaluations of the underlying finite-valued part of the objective function. We show that the proposed method converges (non-asymptotically), under very mild assumptions, close to a stationary point of an appropriate surrogate problem which is related (in a precise mathematical sense) to the original one. This, in turn, provides a new notion of approximate stationarity suitable nonsmooth and nonconvex stochastic composite optimization, generalizing conditions used in the available literature. In light of the generic oracle properties under which the algorithm operates, we showcase the applicability of the approach in a wide range of problems including large classes of two-stage nonconvex stochastic optimization and nonconvex-nonconcave minimax stochastic optimization instances, without requiring convexity of the lower level problems, or even uniqueness of the associated lower level solution maps. We showcase how the developed theory can be applied in each of these cases under general assumptions, providing algorithmic methodologies that go beyond the current state-of-the-art appearing in each respective literature, enabling the solution of problems that are out of reach of currently available methodologies.