ShiftNAS: Towards Automatic Generation of Advanced Mulitplication-Less Neural Networks

TL;DR

ShiftNAS introduces a novel neural architecture search framework specifically designed for multiplication-less neural networks, significantly improving their accuracy and convergence compared to traditional methods.

Contribution

This work pioneers the application of NAS to shift-oriented neural networks, addressing accuracy and convergence issues inherent in previous transfer-based approaches.

Findings

Achieves state-of-the-art accuracy improvements on CIFAR10, CIFAR100, and ImageNet.

Demonstrates better convergence and performance of shift networks.

Sets new benchmarks for multiplication-less neural network performance.

Abstract

Multiplication-less neural networks significantly reduce the time and energy cost on the hardware platform, as the compute-intensive multiplications are replaced with lightweight bit-shift operations. However, existing bit-shift networks are all directly transferred from state-of-the-art convolutional neural networks (CNNs), which lead to non-negligible accuracy drop or even failure of model convergence. To combat this, we propose ShiftNAS, the first framework tailoring Neural Architecture Search (NAS) to substantially reduce the accuracy gap between bit-shift neural networks and their real-valued counterparts. Specifically, we pioneer dragging NAS into a shift-oriented search space and endow it with the robust topology-related search strategy and custom regularization and stabilization. As a result, our ShiftNAS breaks through the incompatibility of traditional NAS methods for bit-shift neural networks and achieves more desirable performance in terms of accuracy and convergence. Extensive experiments demonstrate that ShiftNAS sets a new state-of-the-art for bit-shift neural networks, where the accuracy increases (1.69-8.07)% on CIFAR10, (5.71-18.09)% on CIFAR100 and (4.36-67.07)% on ImageNet, especially when many conventional CNNs fail to converge on ImageNet with bit-shift weights.