NASA: Neural Architecture Search and Acceleration for Hardware Inspired Hybrid Networks

TL;DR

NASA introduces an automated framework combining neural architecture search and hardware acceleration to develop efficient, multiplication-reduced deep neural networks suitable for resource-constrained environments.

Contribution

The paper presents NASA, a novel NAS framework with hardware-inspired operators and a dedicated accelerator, enabling automated design of multiplication-free DNNs with improved efficiency.

Findings

NASA achieves competitive accuracy with multiplication-reduced DNNs.

The co-designed hardware accelerates the execution of NASA-designed networks.

Experimental results validate the efficiency and effectiveness of NASA's approach.

Abstract

Multiplication is arguably the most cost-dominant operation in modern deep neural networks (DNNs), limiting their achievable efficiency and thus more extensive deployment in resource-constrained applications. To tackle this limitation, pioneering works have developed handcrafted multiplication-free DNNs, which require expert knowledge and time-consuming manual iteration, calling for fast development tools. To this end, we propose a Neural Architecture Search and Acceleration framework dubbed NASA, which enables automated multiplication-reduced DNN development and integrates a dedicated multiplication-reduced accelerator for boosting DNNs' achievable efficiency. Specifically, NASA adopts neural architecture search (NAS) spaces that augment the state-of-the-art one with hardware-inspired multiplication-free operators, such as shift and adder, armed with a novel progressive pretrain strategy (PGP) together with customized training recipes to automatically search for optimal multiplication-reduced DNNs; On top of that, NASA further develops a dedicated accelerator, which advocates a chunk-based template and auto-mapper dedicated for NASA-NAS resulting DNNs to better leverage their algorithmic properties for boosting hardware efficiency. Experimental results and ablation studies consistently validate the advantages of NASA's algorithm-hardware co-design framework in terms of achievable accuracy and efficiency tradeoffs. Codes are available at https://github.com/GATECH-EIC/NASA.