On Lower and Upper Bounds in Smooth Strongly Convex Optimization - A Unified Approach via Linear Iterative Methods

TL;DR

This paper introduces a unified framework for analyzing smooth strongly convex optimization algorithms using linear iterative methods, deriving new bounds and providing a natural interpretation of Nesterov's Accelerated Gradient Descent.

Contribution

It develops a novel polynomial-based approach to derive bounds and interpret optimization algorithms, including a fixed-dimensional lower bound for the first time.

Findings

New bounds for optimization algorithms derived via polynomial analysis

A natural interpretation of Nesterov's Accelerated Gradient Descent

Fixed-dimensional lower bounds applicable in practical regimes

Abstract

In this thesis we develop a novel framework to study smooth and strongly convex optimization algorithms, both deterministic and stochastic. Focusing on quadratic functions we are able to examine optimization algorithms as a recursive application of linear operators. This, in turn, reveals a powerful connection between a class of optimization algorithms and the analytic theory of polynomials whereby new lower and upper bounds are derived. In particular, we present a new and natural derivation of Nesterov's well-known Accelerated Gradient Descent method by employing simple 'economic' polynomials. This rather natural interpretation of AGD contrasts with earlier ones which lacked a simple, yet solid, motivation. Lastly, whereas existing lower bounds are only valid when the dimensionality scales with the number of iterations, our lower bound holds in the natural regime where the dimensionality is fixed.