A Closed-Form Bound on the Asymptotic Linear Convergence of Iterative Methods via Fixed Point Analysis

TL;DR

This paper derives a closed-form bound on the asymptotic linear convergence rate of iterative methods, accounting for non-linear effects and approximation errors, providing insights into the convergence behavior beyond the linear regime.

Contribution

It introduces a novel closed-form bound that incorporates first-order approximation errors, enhancing understanding of convergence in iterative algorithms.

Findings

Bound includes linearized iteration count and approximation error overhead

Proven tightness for scalar positively quadratic difference equations

Provides a more comprehensive convergence analysis than traditional fixed-point contraction factors

Abstract

In many iterative optimization methods, fixed-point theory enables the analysis of the convergence rate via the contraction factor associated with the linear approximation of the fixed-point operator. While this factor characterizes the asymptotic linear rate of convergence, it does not explain the non-linear behavior of these algorithms in the non-asymptotic regime. In this letter, we take into account the effect of the first-order approximation error and present a closed-form bound on the convergence in terms of the number of iterations required for the distance between the iterate and the limit point to reach an arbitrarily small fraction of the initial distance. Our bound includes two terms: one corresponds to the number of iterations required for the linearized version of the fixed-point operator and the other corresponds to the overhead associated with the approximation error. With a focus on the convergence in the scalar case, the tightness of the proposed bound is proven for positively quadratic first-order difference equations.