On Convergence of Regularized Barzilai-Borwein Method

TL;DR

This paper proves that the regularized Barzilai-Borwein method converges linearly for strongly convex quadratic functions and introduces an adaptive regularization scheme that enhances its practical performance.

Contribution

The paper establishes R-linear convergence of the RBB method for strongly convex quadratics and proposes an adaptive regularization scheme to improve its efficiency.

Findings

R-linear convergence for strongly convex quadratic functions

Effective adaptive regularization scheme

Numerical verification of improved performance

Abstract

The regularized Barzilai-Borwein (RBB) method represents a promising gradient-based optimization algorithm. In this paper, by splitting the gradient into two parts and analyzing the dynamical system of difference equations governing the ratio of their magnitudes, we establish that the RBB method achieves R-linear convergence for strongly convex quadratic functions of arbitrary dimensions. Specifically, for the two-dimensional case, we provide a concise proof demonstrating that the method exhibits at least R-linear convergence. We propose a simple yet effective adaptive regularization parameter scheme to further improve its performance. A typical numerical example verifies the effectiveness of this scheme.