MultiTASC++: A Continuously Adaptive Scheduler for Edge-Based Multi-Device Cascade Inference

TL;DR

MultiTASC++ is a dynamic scheduler designed for multi-device cascade inference in IoT environments, optimizing throughput, accuracy, and latency across diverse devices and workloads.

Contribution

It introduces a novel adaptive scheduler that manages multi-tenancy in multi-device cascade systems, ensuring high accuracy and efficiency in dynamic IoT settings.

Findings

Maintains targeted satisfaction rate across 100 devices.

Achieves high accuracy with low latency.

Demonstrates scalability and adaptability in diverse environments.

Abstract

Cascade systems, consisting of a lightweight model processing all samples and a heavier, high-accuracy model refining challenging samples, have become a widely-adopted distributed inference approach to achieving high accuracy and maintaining a low computational burden for mobile and IoT devices. As intelligent indoor environments, like smart homes, continue to expand, a new scenario emerges, the multi-device cascade. In this setting, multiple diverse devices simultaneously utilize a shared heavy model hosted on a server, often situated within or close to the consumer environment. This work introduces MultiTASC++, a continuously adaptive multi-tenancy-aware scheduler that dynamically controls the forwarding decision functions of devices to optimize system throughput while maintaining high accuracy and low latency. Through extensive experimentation in diverse device environments and with varying server-side models, we demonstrate the scheduler's efficacy in consistently maintaining a targeted satisfaction rate while providing the highest available accuracy across different device tiers and workloads of up to 100 devices. This demonstrates its scalability and efficiency in addressing the unique challenges of collaborative DNN inference in dynamic and diverse IoT environments.