A Proof of the Exact Convergence Rate of Gradient Descent

TL;DR

This paper precisely determines the worst-case convergence rate of gradient descent for smooth strongly convex functions, confirming prior conjectures and providing exact bounds for optimization performance.

Contribution

It establishes the exact worst-case convergence rate of gradient descent on smooth strongly convex functions, resolving previous conjectures.

Findings

Identifies the smallest possible $ au$ for convergence bounds.

Confirms conjectures by Drori and Teboulle, Taylor et al.

Provides exact convergence rate for gradient descent.

Abstract

We prove the exact worst-case convergence rate of gradient descent for smooth strongly convex optimization on $\mathbb{R}^d$. Concretely, assuming that the objective function $f$ is $\mu$-strongly convex and $L$-smooth, we identify the smallest possible value of $\tau$ for which the inequality $f(x_{N})-f_{*}\leq\tau\|x_{0}-x_{*}\|^{2}$ always holds. The result was previously conjectured by Drori and Teboulle for the case $\mu=0$, and by Taylor, Hendrickx, and Glineur for the case $\mu>0$.