DistrEE: Distributed Early Exit of Deep Neural Network Inference on Edge Devices

TL;DR

DistrEE is a framework that enables distributed deep neural network inference on edge devices with early exit strategies to optimize latency and accuracy trade-offs in resource-constrained environments.

Contribution

It introduces a novel distributed inference framework with early exit policies, combining model early exit and collaborative inference for edge devices.

Findings

Reduces inference latency while maintaining accuracy

Efficiently balances resource usage and inference quality

Demonstrates effectiveness through extensive simulations

Abstract

Distributed DNN inference is becoming increasingly important as the demand for intelligent services at the network edge grows. By leveraging the power of distributed computing, edge devices can perform complicated and resource-hungry inference tasks previously only possible on powerful servers, enabling new applications in areas such as autonomous vehicles, industrial automation, and smart homes. However, it is challenging to achieve accurate and efficient distributed edge inference due to the fluctuating nature of the actual resources of the devices and the processing difficulty of the input data. In this work, we propose DistrEE, a distributed DNN inference framework that can exit model inference early to meet specific quality of service requirements. In particular, the framework firstly integrates model early exit and distributed inference for multi-node collaborative inferencing scenarios. Furthermore, it designs an early exit policy to control when the model inference terminates. Extensive simulation results demonstrate that DistrEE can efficiently realize efficient collaborative inference, achieving an effective trade-off between inference latency and accuracy.