The exact worst-case convergence rate of the gradient method with fixed step lengths for L-smooth functions

TL;DR

This paper precisely characterizes the worst-case convergence rate of the gradient method with fixed step sizes for L-smooth functions, providing exact bounds and optimal step length choices.

Contribution

It introduces a new exact convergence rate for the gradient method with fixed step lengths on L-smooth functions and derives an optimal step length for this bound.

Findings

Established a new exact convergence rate bound.

Identified conditions where the bound is tight.

Derived an optimal fixed step length for the method.

Abstract

In this paper, we study the convergence rate of the gradient (or steepest descent) method with fixed step lengths for finding a stationary point of an $L$-smooth function. We establish a new convergence rate, and show that the bound may be exact in some cases, in particular when all step lengths lie in the interval $(0,1/L]$. In addition, we derive an optimal step length with respect to the new bound.