A predictor-corrector method for the training of deep neural networks

TL;DR

This paper introduces a predictor-corrector training method for deep neural networks that improves training efficiency without sacrificing validation accuracy, demonstrated by a 9% time reduction on CIFAR-10.

Contribution

It proposes a novel predictor-corrector approach for training deep neural nets that maintains accuracy while reducing training time.

Findings

9% training time improvement on CIFAR-10

No loss in validation accuracy

Compatible with standard SGD with backpropagation

Abstract

The training of deep neural nets is expensive. We present a predictor- corrector method for the training of deep neural nets. It alternates a predictor pass with a corrector pass using stochastic gradient descent with backpropagation such that there is no loss in validation accuracy. No special modifications to SGD with backpropagation is required by this methodology. Our experiments showed a time improvement of 9% on the CIFAR-10 dataset.