ShiftAddAug: Augment Multiplication-Free Tiny Neural Network with Hybrid Computation

TL;DR

ShiftAddAug introduces a hybrid approach combining multiplication-free operators with costly multiplication to enhance tiny neural networks' accuracy without inference overhead, validated on image tasks.

Contribution

It proposes a novel hybrid training method and weight sharing technique to improve multiplication-free neural networks' performance.

Findings

Up to 4.95% accuracy increase on CIFAR100.

Outperforms directly trained multiplication-free models.

Effective in image classification and segmentation.

Abstract

Operators devoid of multiplication, such as Shift and Add, have gained prominence for their compatibility with hardware. However, neural networks (NNs) employing these operators typically exhibit lower accuracy compared to conventional NNs with identical structures. ShiftAddAug uses costly multiplication to augment efficient but less powerful multiplication-free operators, improving performance without any inference overhead. It puts a ShiftAdd tiny NN into a large multiplicative model and encourages it to be trained as a sub-model to obtain additional supervision. In order to solve the weight discrepancy problem between hybrid operators, a new weight sharing method is proposed. Additionally, a novel two stage neural architecture search is used to obtain better augmentation effects for smaller but stronger multiplication-free tiny neural networks. The superiority of ShiftAddAug is validated through experiments in image classification and semantic segmentation, consistently delivering noteworthy enhancements. Remarkably, it secures up to a 4.95% increase in accuracy on the CIFAR100 compared to its directly trained counterparts, even surpassing the performance of multiplicative NNs.