Resource-Constrained Edge AI with Early Exit Prediction

TL;DR

This paper introduces an early exit prediction mechanism with a low-complexity Exit Predictor to reduce computation overhead in resource-constrained edge AI, improving inference efficiency and accuracy under varying bandwidths.

Contribution

It proposes a novel Exit Predictor module and a latency-aware extension for early-exit networks, optimizing resource use and performance in edge AI systems.

Findings

Significant reduction in on-device computation overhead.

Improved inference accuracy under bandwidth constraints.

Effective tradeoff between accuracy and efficiency achieved.

Abstract

By leveraging the data sample diversity, the early-exit network recently emerges as a prominent neural network architecture to accelerate the deep learning inference process. However, intermediate classifiers of the early exits introduce additional computation overhead, which is unfavorable for resource-constrained edge artificial intelligence (AI). In this paper, we propose an early exit prediction mechanism to reduce the on-device computation overhead in a device-edge co-inference system supported by early-exit networks. Specifically, we design a low-complexity module, namely the Exit Predictor, to guide some distinctly "hard" samples to bypass the computation of the early exits. Besides, considering the varying communication bandwidth, we extend the early exit prediction mechanism for latency-aware edge inference, which adapts the prediction thresholds of the Exit Predictor and the confidence thresholds of the early-exit network via a few simple regression models. Extensive experiment results demonstrate the effectiveness of the Exit Predictor in achieving a better tradeoff between accuracy and on-device computation overhead for early-exit networks. Besides, compared with the baseline methods, the proposed method for latency-aware edge inference attains higher inference accuracy under different bandwidth conditions.