Worst-case convergence analysis of relatively inexact gradient descent on smooth convex functions

TL;DR

This paper analyzes the worst-case convergence of inexact gradient descent with relative error bounds on smooth convex functions, providing exact and bounded performance estimates using the PEP framework.

Contribution

It introduces a worst-case analysis for relatively inexact gradient descent on smooth convex functions, including exact one-step behavior and bounds for multiple steps.

Findings

Exact worst-case behavior after one step

Computable upper and lower bounds for multiple steps

Guidance on optimal stepsize choice based on convergence rates

Abstract

We consider the classical gradient descent algorithm with constant stepsizes, where some error is introduced in the computation of each gradient. More specifically, we assume some relative bound on the inexactness, in the sense that the norm of the difference between the true gradient and its approximate value is bounded by a certain fraction of the gradient norm. This paper presents a worst-case convergence analysis of this so-called relatively inexact gradient descent on smooth convex functions, using the Performance Estimation Problem (PEP) framework. We first derive the exact worst-case behavior of the method after one step. Then we study the case of several steps and provide computable upper and lower bounds using the PEP framework. Finally, we discuss the optimal choice of constant stepsize according to the obtained worst-case convergence rates.