Regularized methods via cubic model subspace minimization for nonconvex optimization

TL;DR

This paper introduces a novel approach for nonconvex optimization that reuses low-dimensional subspaces in adaptive cubic regularization, leading to significant computational gains while maintaining theoretical complexity guarantees.

Contribution

The paper proposes a method that minimizes cubic models in reused low-dimensional subspaces, combining regularized Newton steps and Krylov subspaces for improved efficiency.

Findings

Significant computational speedups over standard methods.

Preservation of worst-case complexity bounds.

Effective use of polynomial and rational Krylov subspaces.

Abstract

Adaptive cubic regularization methods for solving nonconvex problems need the efficient computation of the trial step, involving the minimization of a cubic model. We propose a new approach in which this model is minimized in a low dimensional subspace that, in contrast to classic approaches, is reused for a number of iterations. Whenever the trial step produced by the low-dimensional minimization process is unsatisfactory, we employ a regularized Newton step whose regularization parameter is a by-product of the model minimization over the low-dimensional subspace. We show that the worst-case complexity of classic cubic regularized methods is preserved, despite the possible regularized Newton steps. We focus on the large class of problems for which (sparse) direct linear system solvers are available and provide several experimental results showing the very large gains of our new approach when compared to standard implementations of adaptive cubic regularization methods based on direct linear solvers. Our first choice as projection space for the low-dimensional model minimization is the polynomial Krylov subspace; nonetheless, we also explore the use of rational Krylov subspaces in case where the polynomial ones lead to less competitive numerical results.