SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives

TL;DR

SAGA is a new incremental gradient optimization algorithm that achieves faster convergence rates, supports composite objectives with proximal operators, and handles non-strongly convex problems adaptively, outperforming previous methods.

Contribution

Introduces SAGA, an incremental gradient method with improved convergence, support for composite objectives, and adaptability to non-strongly convex problems.

Findings

SAGA demonstrates faster convergence in experiments.

Supports composite objectives with proximal operators.

Effective on non-strongly convex problems.

Abstract

In this work we introduce a new optimisation method called SAGA in the spirit of SAG, SDCA, MISO and SVRG, a set of recently proposed incremental gradient algorithms with fast linear convergence rates. SAGA improves on the theory behind SAG and SVRG, with better theoretical convergence rates, and has support for composite objectives where a proximal operator is used on the regulariser. Unlike SDCA, SAGA supports non-strongly convex problems directly, and is adaptive to any inherent strong convexity of the problem. We give experimental results showing the effectiveness of our method.