Early-Exit meets Model-Distributed Inference at Edge Networks

TL;DR

This paper introduces MDI-Exit, a framework combining model-distributed inference with early-exit strategies to reduce computation and communication costs at edge networks, improving efficiency and accuracy.

Contribution

It proposes a novel adaptive framework for early-exit in model-distributed inference, optimizing offloading and data admission at the edge.

Findings

Higher data processing throughput at fixed accuracy.

Improved accuracy for given data rates.

Effective early-exit policy on NVIDIA Nano devices.

Abstract

Distributed inference techniques can be broadly classified into data-distributed and model-distributed schemes. In data-distributed inference (DDI), each worker carries the entire deep neural network (DNN) model but processes only a subset of the data. However, feeding the data to workers results in high communication costs, especially when the data is large. An emerging paradigm is model-distributed inference (MDI), where each worker carries only a subset of DNN layers. In MDI, a source device that has data processes a few layers of DNN and sends the output to a neighboring device, i.e., offloads the rest of the layers. This process ends when all layers are processed in a distributed manner. In this paper, we investigate the design and development of MDI with early-exit, which advocates that there is no need to process all the layers of a model for some data to reach the desired accuracy, i.e., we can exit the model without processing all the layers if target accuracy is reached. We design a framework MDI-Exit that adaptively determines early-exit and offloading policies as well as data admission at the source. Experimental results on a real-life testbed of NVIDIA Nano edge devices show that MDI-Exit processes more data when accuracy is fixed and results in higher accuracy for the fixed data rate.