Training Multi-Layer Binary Neural Networks With Local Binary Error Signals

TL;DR

This paper introduces a fully binary, gradient-free training algorithm for multi-layer Binary Neural Networks that uses local binary error signals, significantly improving accuracy and efficiency over existing methods.

Contribution

It presents the first fully binary, gradient-free training method for multi-layer BNNs using local binary error signals and integer-valued weights, enhancing neurobiological plausibility.

Findings

Up to +35.47% accuracy improvement over single-layer state-of-the-art

Up to +35.30% accuracy over full-precision SGD at same memory cost

Reduces computational cost by 100 to 1000 times

Abstract

Binary Neural Networks (BNNs) significantly reduce computational complexity and memory usage in machine and deep learning by representing weights and activations with just one bit. However, most existing training algorithms for BNNs rely on quantization-aware floating-point Stochastic Gradient Descent (SGD), limiting the full exploitation of binary operations to the inference phase only. In this work, we propose, for the first time, a fully binary and gradient-free training algorithm for multi-layer BNNs, eliminating the need for back-propagated floating-point gradients. Specifically, the proposed algorithm relies on local binary error signals and binary weight updates, employing integer-valued hidden weights that serve as a synaptic metaplasticity mechanism, thereby enhancing its neurobiological plausibility. Our proposed solution enables the training of binary multi-layer perceptrons by using exclusively XNOR, Popcount, and increment/decrement operations. Experimental results on multi-class classification benchmarks show test accuracy improvements of up to +35.47% over the only existing fully binary single-layer state-of-the-art solution. Compared to full-precision SGD, our solution improves test accuracy by up to +35.30% under the same total memory demand, while also reducing computational cost by two to three orders of magnitude in terms of the total number of Boolean gates. The proposed algorithm is made available to the scientific community as a public repository.