Embedded Distributed Inference of Deep Neural Networks: A Systematic Review

TL;DR

This paper systematically reviews recent techniques for distributed inference of neural networks on embedded systems, highlighting current methods, trends, and challenges in deploying AI on resource-constrained devices.

Contribution

It provides a comprehensive taxonomy and analysis of over 100 papers on embedded distributed neural network inference, summarizing recent advances and identifying future challenges.

Findings

Distributed inference enables efficient AI deployment on resource-limited devices.

Edge and cloud integration enhances neural network performance and scalability.

Emerging trends address latency, energy efficiency, and system heterogeneity.

Abstract

Embedded distributed inference of Neural Networks has emerged as a promising approach for deploying machine-learning models on resource-constrained devices in an efficient and scalable manner. The inference task is distributed across a network of embedded devices, with each device contributing to the overall computation by performing a portion of the workload. In some cases, more powerful devices such as edge or cloud servers can be part of the system to be responsible of the most demanding layers of the network. As the demand for intelligent systems and the complexity of the deployed neural network models increases, this approach is becoming more relevant in a variety of applications such as robotics, autonomous vehicles, smart cities, Industry 4.0 and smart health. We present a systematic review of papers published during the last six years which describe techniques and methods to distribute Neural Networks across these kind of systems. We provide an overview of the current state-of-the-art by analysing more than 100 papers, present a new taxonomy to characterize them, and discuss trends and challenges in the field.