Nys-Newton: Nystr\"om-Approximated Curvature for Stochastic Optimization

TL;DR

This paper introduces Nys-Newton, a stochastic optimization method that uses Nyström approximation of the Hessian to improve second-order optimization for large-scale convex and non-convex problems, achieving competitive performance.

Contribution

It proposes a novel Nyström-based Hessian approximation integrated with stochastic gradient methods, reducing computational complexity while maintaining effective curvature information.

Findings

Achieves better Hessian approximation than traditional methods.

Performs competitively with state-of-the-art first-order and quasi-Newton methods.

Provides theoretical convergence analysis for convex functions.

Abstract

Second-order optimization methods are among the most widely used optimization approaches for convex optimization problems, and have recently been used to optimize non-convex optimization problems such as deep learning models. The widely used second-order optimization methods such as quasi-Newton methods generally provide curvature information by approximating the Hessian using the secant equation. However, the secant equation becomes insipid in approximating the Newton step owing to its use of the first-order derivatives. In this study, we propose an approximate Newton sketch-based stochastic optimization algorithm for large-scale empirical risk minimization. Specifically, we compute a partial column Hessian of size ($d\times m$) with $m\ll d$ randomly selected variables, then use the \emph{Nystr\"om method} to better approximate the full Hessian matrix. To further reduce the computational complexity per iteration, we directly compute the update step ($\Delta\boldsymbol{w}$) without computing and storing the full Hessian or its inverse. We then integrate our approximated Hessian with stochastic gradient descent and stochastic variance-reduced gradient methods. The results of numerical experiments on both convex and non-convex functions show that the proposed approach was able to obtain a better approximation of Newton\textquotesingle s method, exhibiting performance competitive with that of state-of-the-art first-order and stochastic quasi-Newton methods. Furthermore, we provide a theoretical convergence analysis for convex functions.