Decentralized Low-Latency Collaborative Inference via Ensembles on the Edge

TL;DR

This paper introduces edge ensembles, a collaborative inference method for edge devices that improves accuracy by sharing quantized features among diverse models, with minimal latency increase.

Contribution

It proposes a novel edge ensemble mechanism enabling multiple edge devices to collaboratively perform DNN inference, reducing communication overhead and enhancing accuracy.

Findings

Edge ensembles improve inference accuracy over local models.

Collaborative inference can outperform larger centralized DNNs.

Minimal latency increase due to communication overhead.

Abstract

The success of deep neural networks (DNNs) is heavily dependent on computational resources. While DNNs are often employed on cloud servers, there is a growing need to operate DNNs on edge devices. Edge devices are typically limited in their computational resources, yet, often multiple edge devices are deployed in the same environment and can reliably communicate with each other. In this work we propose to facilitate the application of DNNs on the edge by allowing multiple users to collaborate during inference to improve their accuracy. Our mechanism, coined {\em edge ensembles}, is based on having diverse predictors at each device, which form an ensemble of models during inference. To mitigate the communication overhead, the users share quantized features, and we propose a method for aggregating multiple decisions into a single inference rule. We analyze the latency induced by edge ensembles, showing that its performance improvement comes at the cost of a minor additional delay under common assumptions on the communication network. Our experiments demonstrate that collaborative inference via edge ensembles equipped with compact DNNs substantially improves the accuracy over having each user infer locally, and can outperform using a single centralized DNN larger than all the networks in the ensemble together.