On Convergence and Stability of Two Extended BB-like Step Sizes

TL;DR

This paper introduces two new extended BB-like step sizes for gradient descent, proving their convergence properties and analyzing their stability, with numerical validation supporting the theoretical findings.

Contribution

It extends the bounds of BB step sizes and analyzes the convergence and stability of the new step sizes in quadratic optimization.

Findings

Both new step sizes achieve R-linear convergence.

The longer step size can be stable under certain conditions.

The shorter step size is generally unstable.

Abstract

The Barzilai-Borwein (BB) step sizes have a profound impact on gradient descent methods. In this work, we propose two new gradient step sizes: one longer than the original long BB step size, and the other shorter than the original short BB step size. This extends the bounds of the original BB step sizes. For strictly convex quadratic optimization problems, we employ the dynamics of difference equations to prove that these two new methods achieve R-linear convergence. Regarding stability, we surprisingly find that under certain conditions, the gradient descent method corresponding to the longer step size is stable, whereas the shorter step size consistently leads to instability. Numerical results validate these stability theories. Here, stability refers to whether the gradient norm decreases monotonically.