Krylov Subspace Descent for Deep Learning

TL;DR

This paper introduces Krylov Subspace Descent, a second order optimization method for deep learning that constructs subspaces using gradients and Hessian approximations, leading to faster convergence and better accuracy than existing methods.

Contribution

The paper presents a novel Krylov subspace-based optimization method that improves convergence speed and accuracy in training deep neural networks, without requiring explicit Hessian matrices or damping parameters.

Findings

Faster convergence than L-BFGS and Hessian Free methods.

Generally better cross-validation accuracy compared to baseline methods.

Simpler and more general than Hessian Free optimization.

Abstract

In this paper, we propose a second order optimization method to learn models where both the dimensionality of the parameter space and the number of training samples is high. In our method, we construct on each iteration a Krylov subspace formed by the gradient and an approximation to the Hessian matrix, and then use a subset of the training data samples to optimize over this subspace. As with the Hessian Free (HF) method of [7], the Hessian matrix is never explicitly constructed, and is computed using a subset of data. In practice, as in HF, we typically use a positive definite substitute for the Hessian matrix such as the Gauss-Newton matrix. We investigate the effectiveness of our proposed method on deep neural networks, and compare its performance to widely used methods such as stochastic gradient descent, conjugate gradient descent and L-BFGS, and also to HF. Our method leads to faster convergence than either L-BFGS or HF, and generally performs better than either of them in cross-validation accuracy. It is also simpler and more general than HF, as it does not require a positive semi-definite approximation of the Hessian matrix to work well nor the setting of a damping parameter. The chief drawback versus HF is the need for memory to store a basis for the Krylov subspace.