Finito: A Faster, Permutable Incremental Gradient Method for Big Data Problems

TL;DR

The paper introduces Finito, an incremental gradient method that achieves four times faster convergence for large smooth strongly convex finite sums and demonstrates superior empirical performance.

Contribution

It presents a novel optimization algorithm with improved theoretical convergence rates and practical sampling strategies for large-scale convex problems.

Findings

Convergence rate four times faster than previous methods

Effective sampling without replacement enhances practical speed

State-of-the-art empirical performance demonstrated

Abstract

Recent advances in optimization theory have shown that smooth strongly convex finite sums can be minimized faster than by treating them as a black box "batch" problem. In this work we introduce a new method in this class with a theoretical convergence rate four times faster than existing methods, for sums with sufficiently many terms. This method is also amendable to a sampling without replacement scheme that in practice gives further speed-ups. We give empirical results showing state of the art performance.