A Stochastic Gradient Method with an Exponential Convergence Rate for Finite Training Sets

TL;DR

This paper introduces a stochastic gradient method with memory that achieves exponential convergence for optimizing strongly convex finite sums, significantly outperforming standard methods in training and test error reduction.

Contribution

The paper presents a novel stochastic gradient algorithm that attains linear convergence by incorporating gradient memory, improving over traditional sublinear methods.

Findings

Achieves exponential convergence rate for strongly convex finite sums.

Outperforms standard stochastic gradient methods in experiments.

Reduces training and test errors more rapidly.

Abstract

We propose a new stochastic gradient method for optimizing the sum of a finite set of smooth functions, where the sum is strongly convex. While standard stochastic gradient methods converge at sublinear rates for this problem, the proposed method incorporates a memory of previous gradient values in order to achieve a linear convergence rate. In a machine learning context, numerical experiments indicate that the new algorithm can dramatically outperform standard algorithms, both in terms of optimizing the training error and reducing the test error quickly.