A note on the optimal convergence rate of descent methods with fixed step sizes for smooth strongly convex functions

TL;DR

This paper provides an elementary analysis of the optimal convergence rates for various descent methods with fixed step sizes on smooth strongly convex functions, extending previous results to more general methods.

Contribution

It introduces a unified convergence analysis framework for general descent methods with fixed step sizes, achieving optimal rates for a broad class of algorithms.

Findings

Optimal convergence rates are established for general descent methods.

The analysis covers variable metric and inexact gradient methods.

Results extend previous work by Taylor, Hendrickx, and Glineur.

Abstract

Based on a result by Taylor, Hendrickx, and Glineur (J. Optim. Theory Appl., 178(2):455--476, 2018) on the attainable convergence rate of gradient descent for smooth and strongly convex functions in terms of function values, an elementary convergence analysis for general descent methods with fixed step sizes is presented. It covers general variable metric methods, gradient related search directions under angle and scaling conditions, as well as inexact gradient methods. In all cases, optimal rates are obtained.