Puzzle: Scheduling Multiple Deep Learning Models on Mobile Device with Heterogeneous Processors

TL;DR

Puzzle is a genetic algorithm-based system that efficiently schedules multiple deep learning models on heterogeneous mobile processors, significantly improving request handling capacity while maintaining real-time performance.

Contribution

It introduces a novel multi-model scheduling approach using genetic algorithms with device-in-the-loop profiling for accurate execution time estimation on heterogeneous mobile hardware.

Findings

Supports 3.7x higher request frequency than baselines

Supports 2.2x higher request frequency than baselines

Maintains real-time performance levels

Abstract

As deep learning models are increasingly deployed on mobile devices, modern mobile devices incorporate deep learning-specific accelerators to handle the growing computational demands, thus increasing their hardware heterogeneity. However, existing works on scheduling deep learning workloads across these processors have significant limitations: most studies focus on single-model scenarios rather than realistic multi-model scenarios, overlook performance variations from different hardware/software configurations, and struggle with accurate execution time estimation. To address these challenges, we propose a novel genetic algorithm-based methodology for scheduling multiple deep learning networks on heterogeneous processors by partitioning the networks into multiple subgraphs. Our approach incorporates three different types of chromosomes for partition/mapping/priority exploration, and leverages device-in-the-loop profiling and evaluation for accurate execution time estimation. Based on this methodology, our system, Puzzle, demonstrates superior performance in extensive evaluations with randomly generated scenarios involving nine state-of-the-art networks. The results demonstrate Puzzle can support 3.7 and 2.2 times higher request frequency on average compared to the two heuristic baselines, NPU Only and Best Mapping, respectively, while satisfying the equivalent level of real-time requirements.