On a Faster $R$-Linear Convergence Rate of the Barzilai-Borwein Method

TL;DR

This paper proves that the Barzilai-Borwein method converges at an $R$-linear rate of $1-1/ ext{condition number}$ for strongly convex quadratic problems, aligning theoretical understanding with empirical success.

Contribution

It establishes a tighter theoretical convergence rate for the BB method on quadratic problems, matching practical observations.

Findings

Proves $R$-linear convergence rate of $1-1/ ext{condition number}$

Constructs an example demonstrating the tightness of the bound

Bridges the gap between empirical performance and theoretical analysis

Abstract

The Barzilai-Borwein (BB) method has demonstrated great empirical success in nonlinear optimization. However, the convergence speed of BB method is not well understood, as the known convergence rate of BB method for quadratic problems is much worse than the steepest descent (SD) method. Therefore, there is a large discrepancy between theory and practice. To shrink this gap, we prove that the BB method converges $R$-linearly at a rate of $1-1/\kappa$, where $\kappa$ is the condition number, for strongly convex quadratic problems. In addition, an example with the theoretical rate of convergence is constructed, indicating the tightness of our bound.