Regularized Step Directions in Nonlinear Conjugate Gradient Methods

TL;DR

This paper introduces a hybrid cubic regularization technique for nonlinear conjugate gradient methods that enhances convergence and efficiency by selectively applying regularization, leading to fewer iterations and faster runtimes.

Contribution

It proposes a novel hybrid cubic regularization approach for CGM, improving convergence and computational efficiency while maintaining memoryless and matrix-free properties.

Findings

Fewer iteration counts compared to standard CGM.

Reduced runtime and fewer resets needed.

Outperforms memoryless BFGS in experiments.

Abstract

Conjugate gradient minimization methods (CGM) and their accelerated variants are widely used. We focus on the use of cubic regularization to improve the CGM direction independent of the step length computation. In this paper, we propose the Hybrid Cubic Regularization of CGM, where regularized steps are used selectively. Using Shanno's reformulation of CGM as a memoryless BFGS method, we derive new formulas for the regularized step direction. We show that the regularized step direction uses the same order of computational burden per iteration as its non-regularized version. Moreover, the Hybrid Cubic Regularization of CGM exhibits global convergence with fewer assumptions. In numerical experiments, the new step directions are shown to require fewer iteration counts, improve runtime, and reduce the need to reset the step direction. Overall, the Hybrid Cubic Regularization of CGM exhibits the same memoryless and matrix-free properties, while outperforming CGM as a memoryless BFGS method in iterations and runtime.