Enhance Curvature Information by Structured Stochastic Quasi-Newton Methods

TL;DR

This paper introduces a structured stochastic quasi-Newton method that efficiently incorporates local curvature information for nonconvex optimization, achieving fast convergence and strong empirical performance.

Contribution

It proposes a novel structured stochastic quasi-Newton approach exploiting Hessian structure for efficient second-order optimization in nonconvex problems.

Findings

Global convergence to stationary points

Local superlinear convergence rate

Competitive performance on deep learning tasks

Abstract

In this paper, we consider stochastic second-order methods for minimizing a finite summation of nonconvex functions. One important key is to find an ingenious but cheap scheme to incorporate local curvature information. Since the true Hessian matrix is often a combination of a cheap part and an expensive part, we propose a structured stochastic quasi-Newton method by using partial Hessian information as much as possible. By further exploiting either the low-rank structure or the kronecker-product properties of the quasi-Newton approximations, the computation of the quasi-Newton direction is affordable. Global convergence to stationary point and local superlinear convergence rate are established under some mild assumptions. Numerical results on logistic regression, deep autoencoder networks and deep convolutional neural networks show that our proposed method is quite competitive to the state-of-the-art methods.