Memory-Efficient Partitioned DNN Inference on Resource-Constrained Android Crowds

TL;DR

This paper introduces CROWDio, a system for efficient DNN inference on resource-limited Android devices by distributing memory load without modifying models, enabling practical edge ML deployment.

Contribution

CROWDio's novel scheduling mechanisms enable large DNN inference across multiple Android devices without model changes, reducing memory and energy usage.

Findings

Achieves peak per-device RSS of 43 MB on DistilBERT

Limits battery draw to 50 mAh per run

Reduces batch latency by 34% with streaming concurrency

Abstract

Deploying large deep neural networks on memory-constrained mobile devices is a central challenge in edge ML. While compression, pruning, and quantization reduce per-parameter cost, transformer-based models remain too large for the 3.3-7.4 GB RAM envelope of commodity Android handsets. We present the DNN pipeline scheduling subsystem of CROWDio, which achieves practical ONNX inference across resource-constrained Android workers without model modification, by distributing memory pressure across devices via five mechanisms: JIT deferred partition loading, a single-partition-resident constraint, a 4-tier affinity scheduler, a zlib-compressed tensor transport, and a streaming 1:1 dependency model. Evaluated on DistilBERT (Sanh et al., 2019) (approximately 67 M parameters, SST-2) across five Android handsets over ten runs, our system holds peak per-device RSS to 43+-2 MB and limits battery draw to 50+-3 mAh per run, while streaming concurrency cuts batch latency 34% below barrier synchronisation.