An experimental comparative study of backpropagation and alternatives for training binary neural networks for image classification

TL;DR

This paper compares backpropagation and alternative training methods for binary neural networks on image classification, aiming to improve training efficiency and model deployment on edge devices.

Contribution

It introduces new experiments on the ImageNette dataset, evaluates three architectures, and adds two novel training alternatives for binary neural networks.

Findings

Binary neural networks can be effectively trained with alternative methods.

Certain architectures perform better with specific training schemes.

Training alternatives improve efficiency and accuracy on edge devices.

Abstract

Current artificial neural networks are trained with parameters encoded as floating point numbers that occupy lots of memory space at inference time. Due to the increase in the size of deep learning models, it is becoming very difficult to consider training and using artificial neural networks on edge devices. Binary neural networks promise to reduce the size of deep neural network models, as well as to increase inference speed while decreasing energy consumption. Thus, they may allow the deployment of more powerful models on edge devices. However, binary neural networks are still proven to be difficult to train using the backpropagation-based gradient descent scheme. This paper extends the work of \cite{crulis2023alternatives}, which proposed adapting to binary neural networks two promising alternatives to backpropagation originally designed for continuous neural networks, and experimented with them on simple image classification datasets. This paper proposes new experiments on the ImageNette dataset, compares three different model architectures for image classification, and adds two additional alternatives to backpropagation.