Efficient Neural Network Training via Subset Pretraining

TL;DR

This paper proposes a subset pretraining method for neural networks, showing that training on small data subsets can achieve results comparable to full dataset training while significantly reducing computational costs.

Contribution

It introduces a novel hypothesis that subset minima approximate full set minima, enabling efficient training with smaller data subsets, validated on multiple image classification benchmarks.

Findings

Subset minima closely approximate full set minima.

Training on small subsets reduces computation to a tenth or less.

Results are comparable to conventional training on full datasets.

Abstract

In training neural networks, it is common practice to use partial gradients computed over batches, mostly very small subsets of the training set. This approach is motivated by the argument that such a partial gradient is close to the true one, with precision growing only with the square root of the batch size. A theoretical justification is with the help of stochastic approximation theory. However, the conditions for the validity of this theory are not satisfied in the usual learning rate schedules. Batch processing is also difficult to combine with efficient second-order optimization methods. This proposal is based on another hypothesis: the loss minimum of the training set can be expected to be well-approximated by the minima of its subsets. Such subset minima can be computed in a fraction of the time necessary for optimizing over the whole training set. This hypothesis has been tested with the help of the MNIST, CIFAR-10, and CIFAR-100 image classification benchmarks, optionally extended by training data augmentation. The experiments have confirmed that results equivalent to conventional training can be reached. In summary, even small subsets are representative if the overdetermination ratio for the given model parameter set sufficiently exceeds unity. The computing expense can be reduced to a tenth or less.