A Momentum Accelerated Adaptive Cubic Regularization Method for Nonconvex Optimization

TL;DR

This paper introduces a momentum-accelerated adaptive cubic regularization method (ARCm) that enhances convergence speed and efficiency for non-convex optimization, outperforming existing methods in practical experiments.

Contribution

The paper develops a new momentum-accelerated adaptive cubic regularization method with proven global and local convergence, improving performance over existing cubic regularization techniques.

Findings

ARCm requires 10-50% fewer iterations than ARC.

ARCm outperforms state-of-the-art cubic regularization methods.

Numerical experiments confirm improved convergence speed.

Abstract

The cubic regularization method (CR) and its adaptive version (ARC) are popular Newton-type methods in solving unconstrained non-convex optimization problems, due to its global convergence to local minima under mild conditions. The main aim of this paper is to develop a momentum-accelerated adaptive cubic regularization method (ARCm) to improve the convergent performance. With the proper choice of momentum step size, we show the global convergence of ARCm and the local convergence can also be guaranteed under the \KL property. Such global and local convergence can also be established when inexact solvers with low computational costs are employed in the iteration procedure. Numerical results for non-convex logistic regression and robust linear regression models are reported to demonstrate that the proposed ARCm significantly outperforms state-of-the-art cubic regularization methods (e.g., CR, momentum-based CR, ARC) and the trust region method. In particular, the number of iterations required by ARCm is less than 10\% to 50\% required by the most competitive method (ARC) in the experiments.