Multilevel Regularized Newton Methods with Fast Convergence Rates

TL;DR

This paper presents novel multilevel regularized Newton methods that leverage coarse models to achieve faster convergence rates for large-scale unconstrained optimization problems, outperforming traditional gradient-based methods.

Contribution

Introduction of regularized multilevel Newton methods that combine coarse models with second-order information for improved convergence in large-scale optimization.

Findings

Provably converges from any initialization.

Achieves convergence rates between Gradient Descent and cubic Newton methods.

Numerical results show significant speed-up over state-of-the-art methods.

Abstract

We introduce new multilevel methods for solving large-scale unconstrained optimization problems. Specifically, the philosophy of multilevel methods is applied to Newton-type methods that regularize the Newton sub-problem using second order information from a coarse (low dimensional) sub-problem. The new \emph{regularized multilevel methods} provably converge from any initialization point and enjoy faster convergence rates than Gradient Descent. In particular, for arbitrary functions with Lipschitz continuous Hessians, we show that their convergence rate interpolates between the rate of Gradient Descent and that of the cubic Newton method. If, additionally, the objective function is assumed to be convex, then the proposed method converges with the fast $\mathcal{O}(k^{-2})$ rate. Hence, since the updates are generated using a \emph{coarse} model in low dimensions, the theoretical results of this paper significantly speed-up the convergence of Newton-type or preconditioned gradient methods in practical applications. Preliminary numerical results suggest that the proposed multilevel algorithms are significantly faster than current state-of-the-art methods.