A Lower Bound for the Optimization of Finite Sums

TL;DR

This paper establishes a fundamental lower bound on the number of iterations needed for algorithms to optimize finite sums of smooth, strongly convex functions, and compares it with existing methods to identify scenarios where new algorithms are beneficial.

Contribution

It introduces a new lower bound for finite sum optimization and analyzes how recent methods compare to this theoretical limit in various data settings.

Findings

Lower bound of (n + \u221a{n(-1)} \, lg(1/\u03b5)) iterations.

Comparison of lower bounds with upper bounds of recent algorithms.

Identification of machine learning scenarios where new methods are advantageous.

Abstract

This paper presents a lower bound for optimizing a finite sum of $n$ functions, where each function is $L$-smooth and the sum is $\mu$-strongly convex. We show that no algorithm can reach an error $\epsilon$ in minimizing all functions from this class in fewer than $\Omega(n + \sqrt{n(\kappa-1)}\log(1/\epsilon))$ iterations, where $\kappa=L/\mu$ is a surrogate condition number. We then compare this lower bound to upper bounds for recently developed methods specializing to this setting. When the functions involved in this sum are not arbitrary, but based on i.i.d. random data, then we further contrast these complexity results with those for optimal first-order methods to directly optimize the sum. The conclusion we draw is that a lot of caution is necessary for an accurate comparison, and identify machine learning scenarios where the new methods help computationally.