Transient growth of accelerated optimization algorithms

TL;DR

This paper analyzes the early transient behavior of accelerated optimization algorithms, revealing how non-normal dynamics cause initial growth and providing bounds on this transient phase based on problem condition numbers.

Contribution

It introduces a linear systems perspective to characterize transient growth in accelerated algorithms and establishes bounds using IQC theory, a novel approach in this context.

Findings

Transient growth is caused by non-normal dynamics.

Transient excursion scales with the square root of the condition number.

Bounds on transient response are tight for large condition numbers.

Abstract

Optimization algorithms are increasingly being used in applications with limited time budgets. In many real-time and embedded scenarios, only a few iterations can be performed and traditional convergence metrics cannot be used to evaluate performance in these non-asymptotic regimes. In this paper, we examine the transient behavior of accelerated first-order optimization algorithms. For convex quadratic problems, we employ tools from linear systems theory to show that transient growth arises from the presence of non-normal dynamics. We identify the existence of modes that yield an algebraic growth in early iterations and quantify the transient excursion from the optimal solution caused by these modes. For strongly convex smooth optimization problems, we utilize the theory of integral quadratic constraints (IQCs) to establish an upper bound on the magnitude of the transient response of Nesterov's accelerated algorithm. We show that both the Euclidean distance between the optimization variable and the global minimizer and the rise time to the transient peak are proportional to the square root of the condition number of the problem. Finally, for problems with large condition numbers, we demonstrate tightness of the bounds that we derive up to constant factors.