Scaling Up Deep Neural Network Optimization for Edge Inference

TL;DR

This paper introduces scalable methods for optimizing deep neural networks for diverse edge devices, enabling efficient DNN inference on mobile and embedded hardware through predictor reuse and neural network-based optimization.

Contribution

The authors propose two novel approaches: reusing performance predictors across devices using monotonicity, and building scalable predictors with a neural optimizer for direct DNN design output.

Findings

Reusing predictors reduces optimization time across multiple devices.

Neural network-based optimizer directly outputs optimal DNN configurations.

Scalable predictors accurately estimate performance metrics for various device-DNN pairs.

Abstract

Deep neural networks (DNNs) have been increasingly deployed on and integrated with edge devices, such as mobile phones, drones, robots and wearables. To run DNN inference directly on edge devices (a.k.a. edge inference) with a satisfactory performance, optimizing the DNN design (e.g., network architecture and quantization policy) is crucial. While state-of-the-art DNN designs have leveraged performance predictors to speed up the optimization process, they are device-specific (i.e., each predictor for only one target device) and hence cannot scale well in the presence of extremely diverse edge devices. Moreover, even with performance predictors, the optimizer (e.g., search-based optimization) can still be time-consuming when optimizing DNNs for many different devices. In this work, we propose two approaches to scaling up DNN optimization. In the first approach, we reuse the performance predictors built on a proxy device, and leverage the performance monotonicity to scale up the DNN optimization without re-building performance predictors for each different device. In the second approach, we build scalable performance predictors that can estimate the resulting performance (e.g., inference accuracy/latency/energy) given a DNN-device pair, and use a neural network-based automated optimizer that takes both device features and optimization parameters as input and then directly outputs the optimal DNN design without going through a lengthy optimization process for each individual device.