QuickNets: Saving Training and Preventing Overconfidence in Early-Exit Neural Architectures

TL;DR

QuickNets is a new training method for neural networks that accelerates training, reduces inference costs, and improves early exit predictions by layer-wise training and confidence-based early exit mechanisms.

Contribution

Introduces QuickNets, a cascaded layer-wise training algorithm with commitment layers to enhance early exits and reduce training and inference costs.

Findings

QuickNets reduce training time compared to standard backpropagation.

Commitment layers improve early exit accuracy by detecting over-confident predictions.

Layer-wise training dynamically allocates learning resources.

Abstract

Deep neural networks have long training and processing times. Early exits added to neural networks allow the network to make early predictions using intermediate activations in the network in time-sensitive applications. However, early exits increase the training time of the neural networks. We introduce QuickNets: a novel cascaded training algorithm for faster training of neural networks. QuickNets are trained in a layer-wise manner such that each successive layer is only trained on samples that could not be correctly classified by the previous layers. We demonstrate that QuickNets can dynamically distribute learning and have a reduced training cost and inference cost compared to standard Backpropagation. Additionally, we introduce commitment layers that significantly improve the early exits by identifying for over-confident predictions and demonstrate its success.