Scaling SNNs Trained Using Equilibrium Propagation to Convolutional Architectures

TL;DR

This paper extends Equilibrium Propagation to convolutional spiking neural networks, demonstrating accurate gradient estimation and state-of-the-art performance on MNIST datasets, highlighting EP's efficiency and biological plausibility.

Contribution

First formulation of training convolutional spiking convergent RNNs with EP, addressing gradient estimation issues and demonstrating competitive results.

Findings

EP enables accurate gradient estimation in spiking RNNs with average pooling.

EP achieves state-of-the-art performance on MNIST and FashionMNIST datasets.

EP offers memory efficiency advantages over BPTT.

Abstract

Equilibrium Propagation (EP) is a biologically plausible local learning algorithm initially developed for convergent recurrent neural networks (RNNs), where weight updates rely solely on the connecting neuron states across two phases. The gradient calculations in EP have been shown to approximate the gradients computed by Backpropagation Through Time (BPTT) when an infinitesimally small nudge factor is used. This property makes EP a powerful candidate for training Spiking Neural Networks (SNNs), which are commonly trained by BPTT. However, in the spiking domain, previous studies on EP have been limited to architectures involving few linear layers. In this work, for the first time we provide a formulation for training convolutional spiking convergent RNNs using EP, bridging the gap between spiking and non-spiking convergent RNNs. We demonstrate that for spiking convergent RNNs, there is a mismatch in the maximum pooling and its inverse operation, leading to inaccurate gradient estimation in EP. Substituting this with average pooling resolves this issue and enables accurate gradient estimation for spiking convergent RNNs. We also highlight the memory efficiency of EP compared to BPTT. In the regime of SNNs trained by EP, our experimental results indicate state-of-the-art performance on the MNIST and FashionMNIST datasets, with test errors of 0.97% and 8.89%, respectively. These results are comparable to those of convergent RNNs and SNNs trained by BPTT. These findings underscore EP as an optimal choice for on-chip training and a biologically-plausible method for computing error gradients.