A Simple Stochastic Variance Reduced Algorithm with Fast Convergence Rates

TL;DR

This paper introduces MiG, a simple stochastic variance reduced algorithm that achieves optimal convergence rates for various convex problems and extends efficiently to sparse and asynchronous settings, with strong theoretical and experimental support.

Contribution

The paper presents MiG, a new simple stochastic variance reduced algorithm with optimal convergence rates, and develops its sparse and asynchronous variants with theoretical analysis.

Findings

MiG attains the best-known convergence rates for convex problems.

Sparse and asynchronous variants of MiG are effective and theoretically sound.

Experiments show practical improvements in machine learning tasks like logistic regression.

Abstract

Recent years have witnessed exciting progress in the study of stochastic variance reduced gradient methods (e.g., SVRG, SAGA), their accelerated variants (e.g, Katyusha) and their extensions in many different settings (e.g., online, sparse, asynchronous, distributed). Among them, accelerated methods enjoy improved convergence rates but have complex coupling structures, which makes them hard to be extended to more settings (e.g., sparse and asynchronous) due to the existence of perturbation. In this paper, we introduce a simple stochastic variance reduced algorithm (MiG), which enjoys the best-known convergence rates for both strongly convex and non-strongly convex problems. Moreover, we also present its efficient sparse and asynchronous variants, and theoretically analyze its convergence rates in these settings. Finally, extensive experiments for various machine learning problems such as logistic regression are given to illustrate the practical improvement in both serial and asynchronous settings.