Deep Neural Network Training without Multiplications

TL;DR

This paper introduces a multiplication-free method for training deep neural networks by replacing floating-point multiplications with addition operations, achieving comparable accuracy to traditional methods without stability issues.

Contribution

It presents a novel approach that eliminates multiplications in neural network training, maintaining accuracy and stability, which is a significant departure from conventional low-precision techniques.

Findings

ResNet trained with addition-based operations achieves competitive accuracy.

The method does not require stability correction techniques.

In some cases, it matches the baseline FP32 accuracy.

Abstract

Is multiplication really necessary for deep neural networks? Here we propose just adding two IEEE754 floating-point numbers with an integer-add instruction in place of a floating-point multiplication instruction. We show that ResNet can be trained using this operation with competitive classification accuracy. Our proposal did not require any methods to solve instability and decrease in accuracy, which is common in low-precision training. In some settings, we may obtain equal accuracy to the baseline FP32 result. This method will enable eliminating the multiplications in deep neural-network training and inference.