Adaptive smoothing mini-batch stochastic accelerated gradient method for nonsmooth convex stochastic composite optimization

TL;DR

This paper introduces AdaSMSAG, an adaptive smoothing mini-batch stochastic accelerated gradient method for nonsmooth convex stochastic composite optimization, improving convergence rates and demonstrating efficiency in portfolio risk management and robust SVM tasks.

Contribution

It proposes a novel adaptive smoothing stochastic gradient method that handles general nonsmooth convex functions without requiring easy proximal operators, with proven convergence and improved complexity.

Findings

Better worst-case iteration complexity than existing methods.

Effective in portfolio risk management applications.

Demonstrates efficiency in Wasserstein distributionally robust SVMs.

Abstract

This paper considers a class of convex constrained nonsmooth convex stochastic composite optimization problems whose objective function is given by the summation of a differentiable convex component, together with a general nonsmooth but convex component. The nonsmooth component is not required to have easily obtainable proximal operator, or have the max structure that the smoothing technique in [Nesterov, 2005] can be used. In order to solve such type problems, we propose an adaptive smoothing mini-batch stochastic accelerated gradient (AdaSMSAG) method, which combines the stochastic approximation method, the Nesterov's accelerated gradient method, and the smoothing methods that allow general smoothing approximations. Convergence of the method is established. Moreover, the order of the worst-case iteration complexity is better than that of the state-of-the-art stochastic approximation methods. Numerical results are provided to illustrate the efficiency of the proposed AdaSMSAG method for a risk management in portfolio optimization and a family of Wasserstein distributionally robust support vector machine problems with real data.