A frequency-domain analysis of inexact gradient methods

TL;DR

This paper uses frequency-domain analysis to study the robustness and convergence rates of inexact gradient methods, including gradient descent, Triple Momentum, and Nesterov's accelerated method, on convex functions.

Contribution

It introduces frequency-domain criteria for analyzing the stability of inexact gradient methods and provides improved convergence bounds for Nesterov's method.

Findings

Frequency-domain criteria effectively analyze robustness of inexact gradient methods.

Improved convergence bounds for Nesterov's accelerated method.

Application to inexact gradient descent and Triple Momentum Method.

Abstract

We study robustness properties of some iterative gradient-based methods for strongly convex functions, as well as for the larger class of functions with sector-bounded gradients, under a relative error model. Proofs of the corresponding convergence rates are based on frequency-domain criteria for the stability of nonlinear systems. Applications are given to inexact versions of gradient descent and the Triple Momentum Method. To further emphasize the usefulness of frequency-domain methods, we derive improved analytic bounds for the convergence rate of Nesterov's accelerated method (in the exact setting) on strongly convex functions.