Minibatch training of neural network ensembles via trajectory sampling

TL;DR

This paper introduces a minibatch trajectory sampling method for efficiently training neural network ensembles, significantly reducing training time and improving inference accuracy on image classification tasks.

Contribution

It presents a novel approach to train neural network ensembles using trajectory sampling with minibatches, achieving substantial computational efficiency gains.

Findings

Training time reduced by two orders of magnitude on MNIST.

Longer trajectories improve inference accuracy.

Method scales with dataset and minibatch size ratio.

Abstract

Most iterative neural network training methods use estimates of the loss function over small random subsets (or minibatches) of the data to update the parameters, which aid in decoupling the training time from the (often very large) size of the training datasets. Here, we show that a minibatch approach can also be used to train neural network ensembles (NNEs) via trajectory methods in a highly efficient manner. We illustrate this approach by training NNEs to classify images in the MNIST datasets. This method gives an improvement to the training times, allowing it to scale as the ratio of the size of the dataset to that of the average minibatch size which, in the case of MNIST, gives a computational improvement typically of two orders of magnitude. We highlight the advantage of using longer trajectories to represent NNEs, both for improved accuracy in inference and reduced update cost in terms of the samples needed in minibatch updates.