Improved SVRG for Non-Strongly-Convex or Sum-of-Non-Convex Objectives

TL;DR

This paper demonstrates that SVRG, originally designed for strongly convex problems, is highly effective and robust for non-strongly convex and sum-of-non-convex objectives, improving theoretical running times.

Contribution

The authors provide new analysis and a novel variant of SVRG that enhance its efficiency in non-strongly convex and sum-of-non-convex settings, broadening its applicability.

Findings

Improved running time bounds for SVRG in non-strongly convex settings.

Enhanced performance of SVRG in sum-of-non-convex objectives like stochastic PCA.

Validation of SVRG's robustness in deep neural network training scenarios.

Abstract

Many classical algorithms are found until several years later to outlive the confines in which they were conceived, and continue to be relevant in unforeseen settings. In this paper, we show that SVRG is one such method: being originally designed for strongly convex objectives, it is also very robust in non-strongly convex or sum-of-non-convex settings. More precisely, we provide new analysis to improve the state-of-the-art running times in both settings by either applying SVRG or its novel variant. Since non-strongly convex objectives include important examples such as Lasso or logistic regression, and sum-of-non-convex objectives include famous examples such as stochastic PCA and is even believed to be related to training deep neural nets, our results also imply better performances in these applications.