A Unified Convergence Theorem for Stochastic Optimization Methods

TL;DR

This paper introduces a unified convergence theorem applicable to various stochastic optimization methods, enabling simplified analysis and extending convergence guarantees to new algorithms in nonconvex, nonsmooth settings.

Contribution

The paper presents a general convergence theorem that applies to multiple stochastic optimization algorithms without algorithm-specific tailoring, including new results for prox-SGD and SMM.

Findings

Unified convergence theorem for stochastic methods

Extended convergence results for SGD and RR

New convergence guarantees for prox-SGD and SMM

Abstract

In this work, we provide a fundamental unified convergence theorem used for deriving expected and almost sure convergence results for a series of stochastic optimization methods. Our unified theorem only requires to verify several representative conditions and is not tailored to any specific algorithm. As a direct application, we recover expected and almost sure convergence results of the stochastic gradient method (SGD) and random reshuffling (RR) under more general settings. Moreover, we establish new expected and almost sure convergence results for the stochastic proximal gradient method (prox-SGD) and stochastic model-based methods (SMM) for nonsmooth nonconvex optimization problems. These applications reveal that our unified theorem provides a plugin-type convergence analysis and strong convergence guarantees for a wide class of stochastic optimization methods.