FreezeOut: Accelerate Training by Progressively Freezing Layers

TL;DR

FreezeOut is a training acceleration technique that progressively freezes layers during training, reducing computation time with minimal or no loss in accuracy across different neural network architectures.

Contribution

The paper introduces FreezeOut, a novel method for speeding up neural network training by gradually freezing layers, which was empirically validated on multiple architectures.

Findings

Up to 20% reduction in training time for DenseNets with slight accuracy loss.

20% speedup for ResNets with no accuracy loss.

No significant speedup observed for VGG networks.

Abstract

The early layers of a deep neural net have the fewest parameters, but take up the most computation. In this extended abstract, we propose to only train the hidden layers for a set portion of the training run, freezing them out one-by-one and excluding them from the backward pass. Through experiments on CIFAR, we empirically demonstrate that FreezeOut yields savings of up to 20% wall-clock time during training with 3% loss in accuracy for DenseNets, a 20% speedup without loss of accuracy for ResNets, and no improvement for VGG networks. Our code is publicly available at https://github.com/ajbrock/FreezeOut