An accelerated first-order method with complexity analysis for solving cubic regularization subproblems

TL;DR

This paper introduces a fast first-order method for solving cubic regularization subproblems by reformulating them into a convex problem, using approximate eigenvalues, and applying accelerated gradient methods, achieving improved complexity.

Contribution

The paper presents a novel reformulation of the cubic regularization subproblem and develops an efficient first-order solution with theoretical complexity guarantees.

Findings

Achieves $ ilde O( ext{epsilon}^{-1/2})$ complexity for approximate solutions.

Outperforms Krylov subspace methods in numerical experiments.

Effective as a subproblem solver in adaptive cubic regularization methods.

Abstract

We propose a first-order method to solve the cubic regularization subproblem (CRS) based on a novel reformulation. The reformulation is a constrained convex optimization problem whose feasible region admits an easily computable projection. Our reformulation requires computing the minimum eigenvalue of the Hessian. To avoid the expensive computation of the exact minimum eigenvalue, we develop a surrogate problem to the reformulation where the exact minimum eigenvalue is replaced with an approximate one. We then apply first-order methods such as the Nesterov's accelerated projected gradient method (APG) and projected Barzilai-Borwein method to solve the surrogate problem. As our main theoretical contribution, we show that when an $\epsilon$-approximate minimum eigenvalue is computed by the Lanczos method and the surrogate problem is approximately solved by APG, our approach returns an $\epsilon$-approximate solution to CRS in $\tilde O(\epsilon^{-1/2})$ matrix-vector multiplications (where $\tilde O(\cdot)$ hides the logarithmic factors). Numerical experiments show that our methods are comparable to and outperform the Krylov subspace method in the easy and hard cases, respectively. We further implement our methods as subproblem solvers of adaptive cubic regularization methods, and numerical results show that our algorithms are comparable to the state-of-the-art algorithms.