Analysis of Gradient Descent with Varying Step Sizes using Integral Quadratic Constraints

TL;DR

This paper uses Integral Quadratic Constraints to analyze gradient descent with variable step sizes, providing convergence guarantees and insights into noise amplification based on problem parameters.

Contribution

It introduces a novel framework modeling gradient descent as a linear parameter-varying system and derives performance bounds for varying step sizes.

Findings

Convergence rate guarantees for step sizes within a known interval.

Recovery of existing bounds for constant step sizes.

Convergence rate depends only on the condition number of the problem.

Abstract

The framework of Integral Quadratic Constraints (IQCs) is used to perform an analysis of gradient descent with varying step sizes. Two performance metrics are considered: convergence rate and noise amplification. We assume that the step size is produced from a line search and varies in a known interval. Modeling the algorithm as a linear, parameter-varying (LPV) system, we construct a parameterized linear matrix inequality (LMI) condition that certifies algorithm performance, which is solved using a result for polytopic LPV systems. Our results provide convergence rate guarantees when the step size lies within a restricted interval. Moreover, we recover existing rate bounds when this interval reduces to a single point, i.e. a constant step size. Finally, we note that the convergence rate depends only on the condition number of the problem. In contrast, the noise amplification performance depends on the individual values of the strong convexity and smoothness parameters, and varies inversely with them for a fixed condition number.