Training Dynamical Binary Neural Networks with Equilibrium Propagation

TL;DR

This paper demonstrates that Equilibrium Propagation can effectively train dynamical binary neural networks, enabling hardware-efficient on-chip learning with minimal accuracy loss compared to full-precision models.

Contribution

It extends EP to train binary weights and activations, achieving comparable accuracy to full-precision models on standard datasets.

Findings

Binary weights with full-precision activations achieve similar accuracy to full-precision models.

Binary weights and activations on MNIST reach within 1% of full-precision accuracy.

The method supports end-to-end training suitable for neuromorphic hardware.

Abstract

Equilibrium Propagation (EP) is an algorithm intrinsically adapted to the training of physical networks, thanks to the local updates of weights given by the internal dynamics of the system. However, the construction of such a hardware requires to make the algorithm compatible with existing neuromorphic CMOS technologies, which generally exploit digital communication between neurons and offer a limited amount of local memory. In this work, we demonstrate that EP can train dynamical networks with binary activations and weights. We first train systems with binary weights and full-precision activations, achieving an accuracy equivalent to that of full-precision models trained by standard EP on MNIST, and losing only 1.9% accuracy on CIFAR-10 with equal architecture. We then extend our method to the training of models with binary activations and weights on MNIST, achieving an accuracy within 1% of the full-precision reference for fully connected architectures and reaching the full-precision accuracy for convolutional architectures. Our extension of EP to binary networks opens new solutions for on-chip learning and provides a compact framework for training BNNs end-to-end with the same circuitry as for inference.