DREAM: A Dynamic Scheduler for Dynamic Real-time Multi-model ML Workloads

TL;DR

DREAM is a novel scheduler designed for real-time multi-model ML workloads that dynamically adapts to workload changes, improving efficiency and reducing energy-delay product on multi-accelerator systems.

Contribution

The paper introduces DREAM, a dynamic scheduler that effectively manages the unique challenges of RTMM workloads by quantifying workload requirements and adapting scheduling decisions accordingly.

Findings

DREAM reduces energy-delay product by up to 97.6% compared to baselines.

DREAM achieves a 50% reduction in UXCost on average across scenarios.

The scheduler effectively handles workload dynamicity and heterogeneity.

Abstract

Emerging real-time multi-model ML (RTMM) workloads such as AR/VR and drone control involve dynamic behaviors in various granularity; task, model, and layers within a model. Such dynamic behaviors introduce new challenges to the system software in an ML system since the overall system load is not completely predictable, unlike traditional ML workloads. In addition, RTMM workloads require real-time processing, involve highly heterogeneous models, and target resource-constrained devices. Under such circumstances, developing an effective scheduler gains more importance to better utilize underlying hardware considering the unique characteristics of RTMM workloads. Therefore, we propose a new scheduler, DREAM, which effectively handles various dynamicity in RTMM workloads targeting multi-accelerator systems. DREAM quantifies the unique requirements for RTMM workloads and utilizes the quantified scores to drive scheduling decisions, considering the current system load and other inference jobs on different models and input frames. DREAM utilizes tunable parameters that provide fast and effective adaptivity to dynamic workload changes. In our evaluation of five scenarios of RTMM workload, DREAM reduces the overall UXCost, which is an equivalent metric of the energy-delay product (EDP) for RTMM defined in the paper, by 32.2% and 50.0% in the geometric mean (up to 80.8% and 97.6%) compared to state-of-the-art baselines, which shows the efficacy of our scheduling methodology.