Leveraging Stochastic Depth Training for Adaptive Inference

TL;DR

This paper introduces a novel adaptive inference method leveraging stochastic depth training, enabling efficient, zero-overhead, and input-dependent model skipping with improved power efficiency and minimal accuracy loss.

Contribution

It proposes using stochastic depth-trained models to select optimal layer-skipping configurations for adaptive inference, simplifying existing techniques.

Findings

Up to 2X power efficiency improvement

Accuracy drops as low as 0.71%

Enhanced resilience to layer-skipping

Abstract

Dynamic DNN optimization techniques such as layer-skipping offer increased adaptability and efficiency gains but can lead to i) a larger memory footprint as in decision gates, ii) increased training complexity (e.g., with non-differentiable operations), and iii) less control over performance-quality trade-offs due to its inherent input-dependent execution. To approach these issues, we propose a simpler yet effective alternative for adaptive inference with a zero-overhead, single-model, and time-predictable inference. Central to our approach is the observation that models trained with Stochastic Depth -- a method for faster training of residual networks -- become more resilient to arbitrary layer-skipping at inference time. We propose a method to first select near Pareto-optimal skipping configurations from a stochastically-trained model to adapt the inference at runtime later. Compared to original ResNets, our method shows improvements of up to 2X in power efficiency at accuracy drops as low as 0.71%.