Approximate Secular Equations for the Cubic Regularization Subproblem

TL;DR

This paper introduces a novel approximate secular equation-based solver for the cubic regularization subproblem in non-convex optimization, reducing computational cost by requiring only some Hessian eigenvalues and demonstrating superior practical performance.

Contribution

It develops and analyzes two approximate secular equations that efficiently solve the CRS with theoretical guarantees, requiring only matrix-vector multiplications.

Findings

The proposed solver outperforms existing methods in experiments.

It provides theoretical bounds on the approximation error.

The method is suitable for high-dimensional problems like deep learning.

Abstract

The cubic regularization method (CR) is a popular algorithm for unconstrained non-convex optimization. At each iteration, CR solves a cubically regularized quadratic problem, called the cubic regularization subproblem (CRS). One way to solve the CRS relies on solving the secular equation, whose computational bottleneck lies in the computation of all eigenvalues of the Hessian matrix. In this paper, we propose and analyze a novel CRS solver based on an approximate secular equation, which requires only some of the Hessian eigenvalues and is therefore much more efficient. Two approximate secular equations (ASEs) are developed. For both ASEs, we first study the existence and uniqueness of their roots and then establish an upper bound on the gap between the root and that of the standard secular equation. Such an upper bound can in turn be used to bound the distance from the approximate CRS solution based ASEs to the true CRS solution, thus offering a theoretical guarantee for our CRS solver. A desirable feature of our CRS solver is that it requires only matrix-vector multiplication but not matrix inversion, which makes it particularly suitable for high-dimensional applications of unconstrained non-convex optimization, such as low-rank recovery and deep learning. Numerical experiments with synthetic and real data-sets are conducted to investigate the practical performance of the proposed CRS solver. Experimental results show that the proposed solver outperforms two state-of-the-art methods.