Failure-Resilient Distributed Inference with Model Compression over Heterogeneous Edge Devices

TL;DR

This paper introduces RoCoIn, a robust distributed inference system for heterogeneous edge devices that uses knowledge distillation and strategic grouping to enhance failure resilience and reduce latency in IoT scenarios.

Contribution

The paper proposes RoCoIn, a novel cooperative inference mechanism employing knowledge distillation and device grouping to improve failure resilience and efficiency in heterogeneous edge environments.

Findings

RoCoIn outperforms baseline methods in inference latency.

It demonstrates high resilience to device failures.

Extensive simulations validate its effectiveness in IoT scenarios.

Abstract

The distributed inference paradigm enables the computation workload to be distributed across multiple devices, facilitating the implementations of deep learning based intelligent services on extremely resource-constrained Internet of Things (IoT) scenarios. Yet it raises great challenges to perform complicated inference tasks relying on a cluster of IoT devices that are heterogeneous in their computing/communication capacity and prone to crash or timeout failures. In this paper, we present RoCoIn, a robust cooperative inference mechanism for locally distributed execution of deep neural network-based inference tasks over heterogeneous edge devices. It creates a set of independent and compact student models that are learned from a large model using knowledge distillation for distributed deployment. In particular, the devices are strategically grouped to redundantly deploy and execute the same student model such that the inference process is resilient to any local failures, while a joint knowledge partition and student model assignment scheme are designed to minimize the response latency of the distributed inference system in the presence of devices with diverse capacities. Extensive simulations are conducted to corroborate the superior performance of our RoCoIn for distributed inference compared to several baselines, and the results demonstrate its efficacy in timely inference and failure resiliency.