Distributed Inference on Mobile Edge and Cloud: A Data-Cartography based Clustering Approach

TL;DR

This paper presents ore, a novel distributed inference method using data cartography to classify sample complexity, enabling resource-efficient DNN deployment across mobile, edge, and cloud platforms with minimal accuracy loss.

Contribution

It introduces ore, a new approach that leverages data cartography to determine sample complexity for optimized distributed inference in resource-constrained environments.

Findings

Reduces inference costs by over 43%

Maintains accuracy drop below 0.5%

Effective across various NLP tasks

Abstract

The large size of DNNs poses a significant challenge for deployment on devices with limited resources, such as mobile, edge, and IoT platforms. To address this issue, a distributed inference framework can be utilized. In this framework, a small-scale DNN (initial layers) is deployed on mobile devices, a larger version on edge devices, and the full DNN on the cloud. Samples with low complexity (easy) can be processed on mobile, those with moderate complexity (medium) on edge devices, and high complexity (hard) samples on the cloud. Given that the complexity of each sample is unknown in advance, the crucial question in distributed inference is determining the sample complexity for appropriate DNN processing. We introduce a novel method named \our{}, which leverages the Data Cartography approach initially proposed for enhancing DNN generalization. By employing data cartography, we assess sample complexity. \our{} aims to boost accuracy while considering the offloading costs from mobile to edge/cloud. Our experimental results on GLUE datasets, covering a variety of NLP tasks, indicate that our approach significantly lowers inference costs by more than 43\% while maintaining a minimal accuracy drop of less than 0.5\% compared to performing all inferences on the cloud. The source code is available at https://anonymous.4open.science/r/DIMEC-1B04.