Stochastic smoothing accelerated gradient method for general constrained nonsmooth convex composite optimization

TL;DR

This paper introduces a stochastic smoothing accelerated gradient method for constrained nonsmooth convex optimization, achieving optimal convergence rates and improving computational efficiency for complex nonsmooth problems.

Contribution

It develops a novel stochastic approximation method that handles a broad class of nonsmooth convex functions without requiring easy proximal computations or linear max structure.

Findings

Achieves optimal iteration complexity of O(1/ε).

Attains optimal SFO complexity of O(1/ε^2).

Demonstrates effectiveness on distributionally robust optimization tasks.

Abstract

We propose a novel stochastic smoothing accelerated gradient (SSAG) method for general constrained nonsmooth convex composite optimization, and analyze the convergence rates. The SSAG method allows various smoothing techniques, and can deal with the nonsmooth term that is not easy to compute its proximal term, or that does not own the linear max structure. To the best of our knowledge, it is the first time to develop a stochastic approximation type method that treats the maximization of finite but numerous nonsmooth convex functions as a stochastic function, which significantly improves the computational efficiency. We prove that the SSAG method can simultaneously achieve the best-known order ${\cal{O}}(\frac{1}{\epsilon})$ of iteration complexity, and the optimal order ${\cal{O}}(\frac{1}{\epsilon^2})$ of $\cal{SFO}$ complexity, using variable sample-size. Numerical results on the application arising from the distributionally robust optimization demonstrate the effectiveness and efficiency of the proposed SSAG method.