Exploring Deep-to-Shallow Transformable Neural Networks for Intelligent Embedded Systems

TL;DR

This paper introduces Double-Win NAS, a neural architecture search method that automatically designs deep networks optimized for accuracy and then transforms them into shallow, hardware-efficient versions for embedded systems.

Contribution

The paper presents a novel NAS paradigm that enables automatic deep-to-shallow network transformation for improved accuracy and efficiency in resource-constrained embedded environments.

Findings

Double-Win NAS outperforms previous NAS methods on embedded hardware.

Transformed shallow networks maintain high accuracy.

Enhanced training techniques improve network elasticity and training performance.

Abstract

Thanks to the evolving network depth, convolutional neural networks (CNNs) have achieved remarkable success across various embedded scenarios, paving the way for ubiquitous embedded intelligence. Despite its promise, the evolving network depth comes at the cost of degraded hardware efficiency. In contrast to deep networks, shallow networks can deliver superior hardware efficiency but often suffer from inferior accuracy. To address this dilemma, we propose Double-Win NAS, a novel deep-to-shallow transformable neural architecture search (NAS) paradigm tailored for resource-constrained intelligent embedded systems. Specifically, Double-Win NAS strives to automatically explore deep networks to first win strong accuracy, which are then equivalently transformed into their shallow counterparts to further win strong hardware efficiency. In addition to search, we also propose two enhanced training techniques, including hybrid transformable training towards better training accuracy and arbitrary-resolution elastic training towards enabling natural network elasticity across arbitrary input resolutions. Extensive experimental results on two popular intelligent embedded systems (i.e., NVIDIA Jetson AGX Xavier and NVIDIA Jetson Nano) and two representative large-scale datasets (i.e., ImageNet and ImageNet-100) clearly demonstrate the superiority of Double-Win NAS over previous state-of-the-art NAS approaches.