An Adaptive Incremental Gradient Method With Support for Non-Euclidean Norms

TL;DR

This paper introduces adaptive variants of the SAGA algorithm with non-Euclidean norm support, providing convergence guarantees and demonstrating competitive performance in large-scale machine learning tasks.

Contribution

It proposes novel adaptive incremental gradient methods with non-Euclidean norms, including a Barzilai-Borwein step-size variant, and extends convergence analysis of SAGA.

Findings

The proposed methods achieve fast convergence in non-Euclidean settings.

Numerical experiments show competitive performance against existing methods.

The analysis supports a broad class of applications with composite objectives.

Abstract

Stochastic variance reduced methods have shown strong performance in solving finite-sum problems. However, these methods usually require the users to manually tune the step-size, which is time-consuming or even infeasible for some large-scale optimization tasks. To overcome the problem, we propose and analyze several novel adaptive variants of the popular SAGA algorithm. Eventually, we design a variant of Barzilai-Borwein step-size which is tailored for the incremental gradient method to ensure memory efficiency and fast convergence. We establish its convergence guarantees under general settings that allow non-Euclidean norms in the definition of smoothness and the composite objectives, which cover a broad range of applications in machine learning. We improve the analysis of SAGA to support non-Euclidean norms, which fills the void of existing work. Numerical experiments on standard datasets demonstrate a competitive performance of the proposed algorithm compared with existing variance-reduced methods and their adaptive variants.