A Simple Practical Accelerated Method for Finite Sums

TL;DR

This paper introduces a simple, practical accelerated optimization method for finite sum problems, improving convergence rates and applicability to non-smooth cases, while maintaining simplicity and minimal parameter tuning.

Contribution

It presents a novel accelerated method based on SAGA that is simpler, requires only one parameter, and extends to non-smooth problems, broadening practical applicability.

Findings

Achieves accelerated convergence on strongly convex smooth problems.

Applicable to non-smooth finite sum problems.

Simpler than existing accelerated methods with minimal parameter tuning.

Abstract

We describe a novel optimization method for finite sums (such as empirical risk minimization problems) building on the recently introduced SAGA method. Our method achieves an accelerated convergence rate on strongly convex smooth problems. Our method has only one parameter (a step size), and is radically simpler than other accelerated methods for finite sums. Additionally it can be applied when the terms are non-smooth, yielding a method applicable in many areas where operator splitting methods would traditionally be applied.