Stochastic Damped L-BFGS with Controlled Norm of the Hessian Approximation

TL;DR

This paper introduces VARCHEN, a stochastic damped L-BFGS algorithm that adaptively estimates Hessian eigenvalues to improve robustness and convergence in nonconvex optimization, especially in deep learning contexts.

Contribution

The paper presents a novel stochastic damped L-BFGS algorithm with eigenvalue-based Hessian approximation control, providing convergence guarantees and empirical robustness improvements.

Findings

VARCHEN outperforms SdLBFGS-VR and SVRG on a nonconvex deep learning problem.

VARCHEN shows comparable performance to existing methods on logistic regression and SVM problems.

The algorithm achieves almost sure convergence with a proven complexity bound.

Abstract

We propose a new stochastic variance-reduced damped L-BFGS algorithm, where we leverage estimates of bounds on the largest and smallest eigenvalues of the Hessian approximation to balance its quality and conditioning. Our algorithm, VARCHEN, draws from previous work that proposed a novel stochastic damped L-BFGS algorithm called SdLBFGS. We establish almost sure convergence to a stationary point and a complexity bound. We empirically demonstrate that VARCHEN is more robust than SdLBFGS-VR and SVRG on a modified DavidNet problem -- a highly nonconvex and ill-conditioned problem that arises in the context of deep learning, and their performance is comparable on a logistic regression problem and a nonconvex support-vector machine problem.