Dimension-Free Iteration Complexity of Finite Sum Optimization Problems

TL;DR

This paper establishes new dimension-free lower bounds for finite sum convex optimization problems, extending the theoretical understanding of the limitations of various first-order and coordinate-descent algorithms.

Contribution

It introduces a framework that provides dimension-free lower bounds, surpassing previous bounds limited by problem dimension and sample size, applicable to many standard optimization methods.

Findings

New dimension-free lower bounds for finite sum optimization.

Applicable to a wide range of algorithms including SAG, SAGA, SVRG, SDCA.

Extends the theoretical limits of first-order and coordinate-descent methods.

Abstract

Many canonical machine learning problems boil down to a convex optimization problem with a finite sum structure. However, whereas much progress has been made in developing faster algorithms for this setting, the inherent limitations of these problems are not satisfactorily addressed by existing lower bounds. Indeed, current bounds focus on first-order optimization algorithms, and only apply in the often unrealistic regime where the number of iterations is less than $\mathcal{O}(d/n)$ (where $d$ is the dimension and $n$ is the number of samples). In this work, we extend the framework of (Arjevani et al., 2015) to provide new lower bounds, which are dimension-free, and go beyond the assumptions of current bounds, thereby covering standard finite sum optimization methods, e.g., SAG, SAGA, SVRG, SDCA without duality, as well as stochastic coordinate-descent methods, such as SDCA and accelerated proximal SDCA.