Training Neural Networks with Stochastic Hessian-Free Optimization

TL;DR

This paper introduces stochastic Hessian-free optimization, combining the benefits of Hessian-free methods and stochastic mini-batch training, enhanced with dropout to prevent overfitting, and demonstrates competitive results on classification and autoencoders.

Contribution

The paper adapts Hessian-free optimization to stochastic mini-batches and incorporates dropout, offering a scalable and effective training method for deep neural networks.

Findings

Achieves competitive performance on classification tasks.

Effective in training deep autoencoders.

Balances between SGD and full Hessian-free methods.

Abstract

Hessian-free (HF) optimization has been successfully used for training deep autoencoders and recurrent networks. HF uses the conjugate gradient algorithm to construct update directions through curvature-vector products that can be computed on the same order of time as gradients. In this paper we exploit this property and study stochastic HF with gradient and curvature mini-batches independent of the dataset size. We modify Martens' HF for these settings and integrate dropout, a method for preventing co-adaptation of feature detectors, to guard against overfitting. Stochastic Hessian-free optimization gives an intermediary between SGD and HF that achieves competitive performance on both classification and deep autoencoder experiments.