On the Convergence Rate of Stochastic Mirror Descent for Nonsmooth Nonconvex Optimization

TL;DR

This paper analyzes the convergence rate of Stochastic Mirror Descent (SMD) in nonconvex, nonsmooth stochastic optimization, establishing a rate of O(1/√t) for convergence to stationary points without mini-batch use.

Contribution

It provides the first non-asymptotic convergence guarantees for SMD in weakly convex, nonsmooth nonconvex problems, including deterministic and proximal variants.

Findings

SMD converges at a rate of O(1/√t) to stationary points.

The analysis applies to both stochastic and deterministic SMD variants.

Efficiency matches existing stochastic subgradient results under stronger stationarity measures.

Abstract

In this paper, we investigate the non-asymptotic stationary convergence behavior of Stochastic Mirror Descent (SMD) for nonconvex optimization. We focus on a general class of nonconvex nonsmooth stochastic optimization problems, in which the objective can be decomposed into a relatively weakly convex function (possibly non-Lipschitz) and a simple non-smooth convex regularizer. We prove that SMD, without the use of mini-batch, is guaranteed to converge to a stationary point in a convergence rate of $ \mathcal{O}(1/\sqrt{t}) $. The efficiency estimate matches with existing results for stochastic subgradient method, but is evaluated under a stronger stationarity measure. Our convergence analysis applies to both the original SMD and its proximal version, as well as the deterministic variants, for solving relatively weakly convex problems.