Energy-Efficient Supervised Learning with a Binary Stochastic Forward-Forward Algorithm

TL;DR

This paper introduces energy-efficient supervised learning algorithms using binary stochastic units and a forward-forward approach, reducing energy consumption and hardware complexity while maintaining competitive performance.

Contribution

It proposes a novel binary stochastic forward-forward algorithm that transforms matrix operations into efficient indexing, enabling hardware-friendly, energy-efficient neural network training.

Findings

Achieves near real-valued forward-forward performance on benchmark datasets.

Estimates about tenfold energy savings compared to traditional methods.

Utilizes p-bits for fast, low-cost binary sampling in hardware implementations.

Abstract

Reducing energy consumption has become a pressing need for modern machine learning, which has achieved many of its most impressive results by scaling to larger and more energy-consumptive neural networks. Unfortunately, the main algorithm for training such networks, backpropagation, poses significant challenges for custom hardware accelerators, due to both its serial dependencies and the memory footprint needed to store forward activations for the backward pass. Alternatives to backprop, although less effective, do exist; here the main computational bottleneck becomes matrix multiplication. In this study, we derive forward-forward algorithms for binary, stochastic units. Binarization of the activations transforms matrix multiplications into indexing operations, which can be executed efficiently in hardware. Stochasticity, combined with tied weights across units with different biases, bypasses the information bottleneck imposed by binary units. Furthermore, although slow and expensive in traditional hardware, binary sampling that is very fast can be implemented cheaply with p-bits (probabilistic bits), novel devices made up of unstable magnets. We evaluate our proposed algorithms on the MNIST, Fashion-MNIST, and CIFAR-10 datasets, showing that its performance is close to real-valued forward-forward, but with an estimated energy savings of about one order of magnitude.