Boosted Dynamic Neural Networks

TL;DR

This paper introduces BoostNet, a novel approach to early-exiting dynamic neural networks inspired by gradient boosting, which addresses training-test mismatch issues and achieves state-of-the-art results on CIFAR100 and ImageNet.

Contribution

The paper proposes BoostNet, a new additive model for EDNNs that mitigates training-test mismatch and improves prediction accuracy through multiple training techniques.

Findings

Achieves state-of-the-art performance on CIFAR100.

Outperforms existing EDNN methods on ImageNet.

Effective in both anytime and budgeted-batch prediction modes.

Abstract

Early-exiting dynamic neural networks (EDNN), as one type of dynamic neural networks, has been widely studied recently. A typical EDNN has multiple prediction heads at different layers of the network backbone. During inference, the model will exit at either the last prediction head or an intermediate prediction head where the prediction confidence is higher than a predefined threshold. To optimize the model, these prediction heads together with the network backbone are trained on every batch of training data. This brings a train-test mismatch problem that all the prediction heads are optimized on all types of data in training phase while the deeper heads will only see difficult inputs in testing phase. Treating training and testing inputs differently at the two phases will cause the mismatch between training and testing data distributions. To mitigate this problem, we formulate an EDNN as an additive model inspired by gradient boosting, and propose multiple training techniques to optimize the model effectively. We name our method BoostNet. Our experiments show it achieves the state-of-the-art performance on CIFAR100 and ImageNet datasets in both anytime and budgeted-batch prediction modes. Our code is released at https://github.com/SHI-Labs/Boosted-Dynamic-Networks.