Understanding the Robustness of Multi-Exit Models under Common Corruptions

TL;DR

This paper investigates how multi-exit models perform under common image corruptions, revealing that distribution shifts impact accuracy, calibration, and exit strategies, and introduces metrics to better understand and improve their robustness.

Contribution

The study provides a comprehensive empirical analysis of MEMs under corruptions, highlighting the effects of distribution shifts and proposing metrics to quantify early-exit behavior.

Findings

Early exits at the first correct layer boost accuracy by 10%.

Realistic early-exit strategies yield only 1% accuracy improvement under corruptions.

Distribution shifts increase misclassification and calibration issues in MEMs.

Abstract

Multi-Exit models (MEMs) use an early-exit strategy to improve the accuracy and efficiency of deep neural networks (DNNs) by allowing samples to exit the network before the last layer. However, the effectiveness of MEMs in the presence of distribution shifts remains largely unexplored. Our work examines how distribution shifts generated by common image corruptions affect the accuracy/efficiency of MEMs. We find that under common corruptions, early-exiting at the first correct exit reduces the inference cost and provides a significant boost in accuracy ( 10%) over exiting at the last layer. However, with realistic early-exit strategies, which do not assume knowledge about the correct exits, MEMs still reduce inference cost but provide a marginal improvement in accuracy (1%) compared to exiting at the last layer. Moreover, the presence of distribution shift widens the gap between an MEM's maximum classification accuracy and realistic early-exit strategies by 5% on average compared with the gap on in-distribution data. Our empirical analysis shows that the lack of calibration due to a distribution shift increases the susceptibility of such early-exit strategies to exit early and increases misclassification rates. Furthermore, the lack of calibration increases the inconsistency in the predictions of the model across exits, leading to both inefficient inference and more misclassifications compared with evaluation on in-distribution data. Finally, we propose two metrics, underthinking and overthinking, that quantify the different behavior of practical early-exit strategy under distribution shifts, and provide insights into improving the practical utility of MEMs.