OmniBoost: Boosting Throughput of Heterogeneous Embedded Devices under Multi-DNN Workload

TL;DR

OmniBoost is a lightweight multi-DNN management system that significantly improves throughput on heterogeneous embedded devices by using stochastic exploration and performance estimation, effectively utilizing diverse accelerators.

Contribution

The paper introduces OmniBoost, a novel multi-DNN controller that enhances throughput by efficiently exploring solution space in heterogeneous embedded systems.

Findings

Achieves an average 4.6x throughput boost over state-of-the-art methods.

Effectively utilizes heterogeneous accelerators in embedded systems.

Validated on HiKey970 development board.

Abstract

Modern Deep Neural Networks (DNNs) exhibit profound efficiency and accuracy properties. This has introduced application workloads that comprise of multiple DNN applications, raising new challenges regarding workload distribution. Equipped with a diverse set of accelerators, newer embedded system present architectural heterogeneity, which current run-time controllers are unable to fully utilize. To enable high throughput in multi-DNN workloads, such a controller is ought to explore hundreds of thousands of possible solutions to exploit the underlying heterogeneity. In this paper, we propose OmniBoost, a lightweight and extensible multi-DNN manager for heterogeneous embedded devices. We leverage stochastic space exploration and we combine it with a highly accurate performance estimator to observe a x4.6 average throughput boost compared to other state-of-the-art methods. The evaluation was performed on the HiKey970 development board.