EH-DNAS: End-to-End Hardware-aware Differentiable Neural Architecture Search

TL;DR

EH-DNAS introduces an end-to-end differentiable approach that integrates hardware benchmarking to optimize neural architectures for various platforms, significantly improving hardware efficiency without sacrificing accuracy.

Contribution

It presents a novel end-to-end hardware-aware DNAS framework with a differentiable hardware performance predictor and benchmarking tool for customized accelerators.

Findings

Achieves 1.4x hardware performance improvement on customized accelerators.

Achieves 1.6x hardware performance improvement on existing hardware.

Maintains classification accuracy while optimizing for hardware efficiency.

Abstract

In hardware-aware Differentiable Neural Architecture Search (DNAS), it is challenging to compute gradients of hardware metrics to perform architecture search. Existing works rely on linear approximations with limited support to customized hardware accelerators. In this work, we propose End-to-end Hardware-aware DNAS (EH-DNAS), a seamless integration of end-to-end hardware benchmarking, and fully automated DNAS to deliver hardware-efficient deep neural networks on various platforms, including Edge GPUs, Edge TPUs, Mobile CPUs, and customized accelerators. Given a desired hardware platform, we propose to learn a differentiable model predicting the end-to-end hardware performance of neural network architectures for DNAS. We also introduce E2E-Perf, an end-to-end hardware benchmarking tool for customized accelerators. Experiments on CIFAR10 and ImageNet show that EH-DNAS improves the hardware performance by an average of $1.4\times$ on customized accelerators and $1.6\times$ on existing hardware processors while maintaining the classification accuracy.