Learn&Drop: Fast Learning of CNNs based on Layer Dropping

TL;DR

This paper introduces a novel training method for CNNs that reduces forward pass computations during training by dynamically dropping layers based on parameter change scores, significantly speeding up training.

Contribution

The method uniquely focuses on decreasing training forward operations through layer dropping, validated on VGG and ResNet architectures with substantial speed improvements.

Findings

Training time more than halved without significant accuracy loss

FLOPs reduction in forward pass ranges from 17.83% to 83.74%

Effective on MNIST, CIFAR-10, and Imagenette datasets

Abstract

This paper proposes a new method to improve the training efficiency of deep convolutional neural networks. During training, the method evaluates scores to measure how much each layer's parameters change and whether the layer will continue learning or not. Based on these scores, the network is scaled down such that the number of parameters to be learned is reduced, yielding a speed up in training. Unlike state-of-the-art methods that try to compress the network to be used in the inference phase or to limit the number of operations performed in the backpropagation phase, the proposed method is novel in that it focuses on reducing the number of operations performed by the network in the forward propagation during training. The proposed training strategy has been validated on two widely used architecture families: VGG and ResNet. Experiments on MNIST, CIFAR-10 and Imagenette show that, with the proposed method, the training time of the models is more than halved without significantly impacting accuracy. The FLOPs reduction in the forward propagation during training ranges from 17.83\% for VGG-11 to 83.74\% for ResNet-152. These results demonstrate the effectiveness of the proposed technique in speeding up learning of CNNs. The technique will be especially useful in applications where fine-tuning or online training of convolutional models is required, for instance because data arrive sequentially.