A new Barzilai-Borwein steplength from the viewpoint of total least squares

TL;DR

This paper introduces a novel Barzilai-Borwein steplength derived from total least squares, bridging the gap between existing BB steplengths based on ordinary and data least squares, enhancing gradient descent methods.

Contribution

The paper proposes a new BB steplength based on total least squares, providing a unified approach that improves upon existing steplength choices in gradient descent.

Findings

The new steplength lies between the two existing BB steplengths.

It offers a more balanced and potentially more effective step size.

The approach connects least squares concepts to optimize gradient methods.

Abstract

Barzilai-Borwein (BB) steplength is a popular choice in gradient descent method. By observing that the two existing BB steplengths correspond to the ordinary and the data least squares, respectively, we employ the third kind of least squares, the total least squares, to create a new BB steplength, which is shown to lie between the two existing BB steplengths.