DynaMIX: Resource Optimization for DNN-Based Real-Time Applications on a Multi-Tasking System

TL;DR

DynaMIX is a dynamic resource optimization framework for DNN-based real-time applications on autonomous vehicles, balancing accuracy and latency by adaptive model reconfiguration to meet safety-critical timing constraints.

Contribution

It introduces a novel optimization approach and dynamic model reconfiguration technique to improve resource utilization and performance of concurrent DNN applications in real-time systems.

Findings

Effective constraint satisfaction under various conditions

Improved inference accuracy with resource optimization

Feasible dynamic reconfiguration for real-time deployment

Abstract

As deep neural networks (DNNs) prove their importance and feasibility, more and more DNN-based apps, such as detection and classification of objects, have been developed and deployed on autonomous vehicles (AVs). To meet their growing expectations and requirements, AVs should "optimize" use of their limited onboard computing resources for multiple concurrent in-vehicle apps while satisfying their timing requirements (especially for safety). That is, real-time AV apps should share the limited on-board resources with other concurrent apps without missing their deadlines dictated by the frame rate of a camera that generates and provides input images to the apps. However, most, if not all, of existing DNN solutions focus on enhancing the concurrency of their specific hardware without dynamically optimizing/modifying the DNN apps' resource requirements, subject to the number of running apps, owing to their high computational cost. To mitigate this limitation, we propose DynaMIX (Dynamic MIXed-precision model construction), which optimizes the resource requirement of concurrent apps and aims to maximize execution accuracy. To realize a real-time resource optimization, we formulate an optimization problem using app performance profiles to consider both the accuracy and worst-case latency of each app. We also propose dynamic model reconfiguration by lazy loading only the selected layers at runtime to reduce the overhead of loading the entire model. DynaMIX is evaluated in terms of constraint satisfaction and inference accuracy for a multi-tasking system and compared against state-of-the-art solutions, demonstrating its effectiveness and feasibility under various environmental/operating conditions.