Asymptotic behaviour of a family of gradient algorithms in R^d and Hilbert spaces

TL;DR

This paper analyzes the long-term behavior of a family of gradient algorithms, including steepest descent and minimum residues, in finite and infinite-dimensional spaces, revealing they share similar convergence properties and attractors.

Contribution

It generalizes previous results by showing all algorithms in the family have the same asymptotic behavior and extends the analysis to Hilbert spaces.

Findings

All algorithms converge to a two-point attractor.

They exhibit similar asymptotic convergence rates.

The stability of the attractor is thoroughly analyzed.

Abstract

The asymptotic behaviour of a family of gradient algorithms (including the methods of steepest descent and minimum residues) for the optimisation of bounded quadratic operators in R^d and Hilbert spaces is analyzed. The results obtained generalize those of Akaike (1959) in several directions. First, all algorithms in the family are shown to have the same asymptotic behaviour (convergence to a two-point attractor), which implies in particular that they have similar asymptotic convergence rates. Second, the analysis also covers the Hilbert space case. A detailed analysis of the stability property of the attractor is provided.